Todd Gardner: JavaScript Forensics | JSConf EU 2015


Hi everybody. This
is JavaScript forensics. It’s about your awesome web-applications and how they completely fail
for your end users. It’s about diagnosing and debugging production web-applications.
In a lot of ways, we’re still running in the wild, wild web. There are dozens of tools
we use to build our applications and we combine that with all the different devices and browsers
and plug-ins that our end users use when visiting it, over a network we don’t control and is
often unpredictable, that leaves a lot of room for JavaScript out laws, that’s what
we are going to talk about today. Five of the most common JavaScript at laws that rage
havoc on applications, we are going to look at what information we can capture about them,
how do we diagnose them and how do we fix them. To do that we need to lure in the out
laws, here is what we are going to do that, it’s a little app called soliloquy, any venture
capitalists, get your cheques ready. It’s what I call Anand Vemuri social network,
it’s kind of like Twitter, it let’s you type in status up dates like Twitter , an awesome
timeline of events like Twitter, awesome in line adds like Twitter, some fantastic sponsors
like cats, Hasselhoff, bears – {Laughter}, but it leaves off the most irritating part
about Twitter, other people! {Laughter} . Because with soliloquy, it’s only about you,
only you can write up dates only you can see your up dates, obviously it’s going to be
huge, like next year, millions and millions of dollars. So, I need to protect it, as I’m
building it out this thing is going to be big, I need to make sure my users have a great
experience, I’m going to be watching how they are reacting with t, to do that, a little
application helped work on, track JS, a JavaScript error monitoring tool, with that let’s dig
into the first errors. Here are the first ones, scripty Jo, it comes
in as script error, hiding the application hard, it hits really hard in the first week
I launched soliloquy, I had one 400 errors coming in, it’s not limited one thing, hitting
everything. So let’s look at it and debug this ands’ if we can understand what exactly
is going on. If we pop open the debugger console and start
looking at this, when we load it we see two errors in our console right away, we see we
were able to record a message, script error, but Chrome nicely can tell me all sorts of
other information about it, the Chrome debugger is independent of the constraints of a JavaScript
execution environment, it can tell me real information, programmatically, what I was
able to capture from the browsing session and pull back to my server to see the errors,
is a very unhelpful message, “Script error”, with no file, no line, no column and no stack
trace. That’s hard, we don’t even know where to start with this, it’s like something blew
up swear, we don’t know how. How did this come to exist? Chrome nicely told us that
this, the real error involved here this reference error from side bar add provider. However,
I want to be able to know what the real errors are when we are interacting with this, let’s
look at the side bar add provider. What’s interesting about this, it’s not necessarily
what is in this script, but where it comes from. The side bar add provider is being loaded
from adds.local, if you see in the very small upper corner, it’s being loaded from local
host, it’s being served from a different domain and now subject to the same original policy
protections on the browser. In the last presentation we learned about
calls for posting requests to different end points, we can apply some of the same things
here, so because we are loading the side bar add provider from a different domain the browser
is saying, “Hey, I don’t trust that”, so when it blows up it obfuscates the error. It trusts
it enough to get the error message, to do that we need to serve the add bar side provider
with the errors. If we look at how it’s getting served… I’ve already added these from my
advertising provider has already added to these to the network and the script server
says it can be loaded from anywhere with the wild card, I now need to teach Chrome, load
it and give the same policy. Let’s a look at the code and see how it works, here’s the
main document, I trust everybody had their NDAs in order. Here at the bottom is where I’m loading all
of my JavaScript, here’s the side bar add provider, here’s the very last one, adding
from ASDA local, none of my scripts are coming from my origin, none of them are coming from
local host, I’m letting my scripts from a CDN to improve performance and so this is
going to be a problem persuasive across my application. So I can teach the browser that
I want to go ahead and trust all of these scripts, as long as those scripts are served
with the cross origin headers we can get rid of this error, so we can decorate the script
elements with the attribute cross origin and it will tell newer browsers to go ahead and
look for the CORS errors. You will see that we actually haven’t fixed
the error yet, but we have allowed us to do is capture real information about this JavaScript
error. Now, what ever we are able to pull out from our real browser sessions has real
information that we can use to aggregate and analyse our data, we can see that this was
a uncaught error, from my side bar add provider with a full stack trace, now we dent have
to deal with script error any more. So this was scripty Jo, it’s browser obfuscation
you well have to deal with this right away if you track client side errors, if you load
some of the CTN, it conceals all the real problems your application has, now we have
this one out of the way, let’s move on to another one. Jane Adsy, the actual underlying
area, the get random adds is not defined. What’s odd about Jane Adsy, I didn’t change
soliloquy, it has been stable, I didn’t want to change it before the conference, all of
a Sutton yesterday it blew up, no change from me at all, I had tonnes and tonnes of errors,
I don’t know how that could be. So let’s take a look at it again and see if we can figure
out what happened: we see the uncaught reference, get random add is not defined, we can navigate
directly to the file and take a look at what’s going on. So this, again, is from my third party, I
do not own this script I just bring it in because I pay the side bar add provider, or
they pay me to place adds on their site, or my site. So they are
serving this JavaScript that’s calling function, get random add and it’s not there. If we look
around in the file we see that there is another function a bit further down, called get random
adds, maybe somebody dropped an s, when they were trying to save a file. Could be we just had a fat finger situation
with the advertising provider, when we bring up a local copy we see that there is these
two calls to get random add, although it doesn’t exist. If we pull a local copy of this file
and save it without the stray character in it… check-out all the awesome new adds that
I now have on it, for more bears, bacon and Hasselhoff. So Jane third party adsy, can cause errors
in your application any time you bring in a third party. This is something we don’t
often recognise, we’re usually just trying to build the software we are trying to build
for are customers, we bring in social networks, advertising, payment processors, all kinds
of really valuable third party tools but we are giving them the ability to run JavaScript
on our application and sometimes to change the JavaScript without telling us, it can
cause all kinds of uncontrolled changes in your environment. So, the next error is Susan scopes, it’s cannot
read property destroy of undefined, a major global error not happening to everybody, we
see a strong correlation, every time a user is trying to delete something out of their
timeline we see we get this error in the console, or we get this error reported shortly thereafter.
We can take a look at it again and receive we can recreate the situation. So the error report that we get is that the
user tries to delete one of the statements and they can’t, it loads up for them. So, let’s open our console back up, we will
try to delete the statement, we saw console messages printed out saying trying to delete
the statement. Then we recorded, “Cannot read property, destroy of undefined”, just like
we saw in the tool. Chrome nicely reported as well, it’s telling us this is coming from
statement ujs line 30, let’s gig into a little bit more, pop over the stack trace there,
it’s awesome, there was an anonymous function on that line, something blew up there, we
don’t really have any context. So Chrome, about six or eight months ago, shipped a really
cool thing that’s probably my favourite thing they have ever done, it’s this little check
box right here in the ‘Sources’ tab, it’s hidden away and you wouldn’t notice it if
you didn’t look for it, if you turn it on, it will let us get over the big stack trace
limitation in JavaScript, because JavaScript is async and is usually responding to an event,
the step trace can only go back to the place where that event started, I don’t know what
attached that click handler, or started that, or created an AJAX fault, I know what it up,
Chrome has access – so if we turn this on and reload, let’s go ahead and delete another
statement, blew up again, just like we expected. Now if we check-out the stack trace, there
is all kinds or more awesomeness here, anonymous function on statement view line 30, but that
was passed to a set Time Out, it was called by a function called ‘On delete’, called in
response to jquery events that seems to make sense with the actions I took, I clicked the button, vent thrown,
caught it on delete, then passing into set Time Out, let’s look at statement of UJS and
see what’s happening. Here’s line 30, of are on delete, right away
here’s the code smell, it’s we’re passing functional argument in and using the word,
“This”,. This is obviously the most common error that we all seem to make over hand over
and over again, parsing a value when it’s going to change context over it’s lifetime.
There is a couple of ways to fix this, the comment, the most obvious ones are either
var that=this, or var self=this. Which, you know, depends on which JavaScript book
you read first is your preference! {Laughter} It might be a little bit more obvious because
we’re only doing this one thing, it’s to say var model equals this.model and simple reference
it as model. Or if we want to be super cool and hip, we could use the new ways of doing
things and use a bind, and chain.bind this to the end of that function. Which will glue
the value of this from the outer context into the value of this on the inner context. That’s
pretty cool, except it won’t work quite everywhere. I want Soliloquy to be super compatible. So
instead of doing that maybe I will use a utility. I will bring in underscore, low dash or whatever
your preference is there and use it’s bind method to tie the 2 together. This will be
super compatible it will let me go back all the way to IE 8 or 7 support for it. Now we have actually bound up this thing together
we can try to delete stuff again. Now we can delete to our hearts content. Awesome. So, the Susan, this and that Scopes happens
any time we get a functional argument and usually manifest itself as something is not
defined. This will happen any time you are using you’re parsing a functional argument
in, either using call backs or promises or anything like that. Now typically you could
find these sorts of errors before they made it out into your production environment, there
is a finite set of things that can go wrong here. However JavaScript apps once they reach
a certain size get difficult to test every possible condition. I still see this in production
a whole lot. All right. Logan No Load’em. Inline ads
it’s not a function. So we saw this error start coming up. It doesn’t happen to everybody,
but it happens to a lot of people. We saw Soliloquy gets 2500 hits a week but we see
about 130 people encounter this error, this a.inline ads is not a function. So it’s not
everybody, it always seems to happen like right away when it does happen so let’s see
if we can understand why that would exist. So let’s pop back into Soliloquy. So when
I just load my app, it obviously is not throwing that error. Doesn’t happen for me here on
this network. So let’s look in our code for where is inline ads even being defined. If
we search Soliloquy for inline ads. There’s 2 places, the first is in the inline ad provider.
The inline ad provider is another advertising network I use to make some money for Soliloquy
it defines this API, the window.inline ads will be a function that I will pass stuff
into. There in my main function, my main initialisation I try to set up my ads Soliloquy I use the
API they define for me this window.inline ads I pass in this information about what
I want. So it should never be undefined, window.inline
ads it’s called here inside of a ready function, we’re waiting until the DOM is done, and the
inline ad provider is loaded here just directly from a script, so I cannot imagine why this
wouldn’t work. We load it doesn’t happen. So, let’s see what would it manifest itself
if it didn’t load. Let just comment this out. Let’s say hey maybe somebody is striping that
out. Here’s definitely the error we’re seeing reported. A.inline ads is into the function,
but look at the experience that my end user got when they encountered this problem. It’s
not just like the inline ads didn’t load, this nothing loaded. The status list didn’t
load the inline ads, the sidebar ads didn’t load if they try and type a message, it just
fails. The app is completely broken for them when this issue happens to happen. So looking into this, this was one of the
things that surprised me most understanding production JavaScript debugging is that sometimes
due to craziness of online networks, one script will fail to load in your application. Just
one. Maybe it’s DNS look up failed because you are changing cell networks, or maybe they
had a blip in their CDN, and so we have seen numerous cases where a single script in your
application fails to load and so depending on what script that is, it can vastly, you
can end up in a vastly different situation in the case of Soliloquy catastrophic this
app failed to load entirely, maybe you are bringing in a payment processor, PayPal or
Braintree to do your work. What if their script fails to load because of a network blip? What
state does that leave your credit card form in when their script just is not on the page?
So here in my code in Soliloquy, I wasn’t being very safe and recognising that this
situation could happen. Because I am just calling window.online ads directly as my main
execution. So when this doesn’t exist, it stops, set up ads doesn’t work, sidebar ad
fails to work, when we return from set up ads I never fetch my main statement list,
I never do the main things as part of Soliloquy. So what we need to do is we need to be more
cautious about the APIs that we use before we use them. Just because it’s the script
is supposed to be there doesn’t mean it will always be there. So we can do this real simple,
just a check. Let’s see if window.inline ads is a thing before we use, if it’s a thing
we’ll go ahead and call it. If it’s not a thing, maybe we should log something.
We should a complaint to our inline ad provider see if we can get a discount next time. So now the real simple check in place, the
experience from my end user is much better, it’s still not great for me because I am not
getting impressions on this ad, my inline ads are not there but the end user doesn’t
really notice. Would you really notice if Twitter loaded and there weren’t any inline
ads anymore? So Logan, missing script. This happens all
kinds of reason, flaky networks, poor network infrastructure, maybe a new deployment is
happening right when a request is being made. There are infinite edge cases here that can
cause a script to fell to load on the internet. As you are bringing in scripts, third parties,
always verify the script exist successfully before you use it the wise an error will cause
a catastrophic change that will stop all future execution. This is the error I will talk about today
this my browser crashing. My browser crashing happens we’re seeing reports of slowing performance
of Soliloquy. narcissistically writing all their thoughts into soliloquy, after hours
of typing they tell us their browser has crashed. There’s no good way to record the browser
crashed in JavaScript, because the browser crashed. So we can’t really get an AJAX call
out the door before it crashes. So we have to rely on the old school way of figuring
out your reports, people complaining on the internet. The worse is they are complaining
probably on Twitter which is the thing we’re trying to get rid off. Let take a look at Soliloquy, see if we can
identify why this happens, slowing performance crashing the browser. Sounds like maybe we
have a memory leak, let’s look to see if we have a memory leak. Oh let’s fix our inline
ads provider, probably want that loading again. So we’re going to check out a memory leak.
There’s a couple of different ways we can too this my personal favourite is to check
out the timeline. Now the timeline is one of the newer tools in the Chrome debugger,
and it let’s us capture all kinds of interesting state changes as our application is running.
So what I am going to do here I am going to type a few status into Soliloquy, then I will
delete them back out which will hopefully return me to same state my application was
in before. Hopefully, the same level of memory utilisation and if it doesn’t, that could
indicate a leak. So I am going to go ahead and start recording
here. I am going to type in some messages. Soliloquy also has this awesome ‘I am feeling
random’ button which will generate a random series of characters an stick it in. It’s
great for the creativity. So I created 3 I will delete these 3 back out. Then let’s take a look and what we captured.
All right so there’s a lot going on here. There’s a lot going on here. It’s hard to
know where to start. So, I like to start down here in this main thread where we see in blue
the total amount of JavaScript memory that’s being used and in orange, golden rod whatever
, the total number of JavaScript listeners. And we see ingredient number of nodes. If
we just look at those things, we see a pattern happening here. It’s here at the beginning
we see it a tick up in all 3, and tick us up again, and again, and again, it looks like
we clean a little bit up, then it keeps going right back up. At no point did we back out,
at no point did we really clean up back to our original state. So this is not really enough to go off of,
so we need to understand a little bit more. So what we’re going to do is before I really
figured out how to become productive with my life, I spent a lot of time playing video
games particularly old Counterstrike. We’re going to use classic Counterstrike controls
to fly into this chart and understand more what happens so yet just WSAD, standard first
person shooter controls, we’ll start flying into these different reports we can start
seeing some information about them. So, as we get closer, we see that this step
is not one thing, but several little things. We can strafe round here we can just left,
right make sure we dodging and weaving, make sure the memory leaking bullets are not hitting us. That’s not the one I want either.
We will have to strafe again. Here is an interesting one, there we go that’s the one I want. So now we’re going to fly in here even closer,
we’re going to see more sufficient about here and as we dig in we can start seeing really
good detail about the timeline and what’s actually happening, causing these changes
in memory leaks in documents. We see that this call is stemming from a timer in sidebarAdProvider
that’s causing a memory to tick up, and again, a timer in a sidebarAdProvider causing a memory
to tick up. This is giving us a good invitation of where our memory leak might be. So if we take a look at the sidebarAdProvider,
we can see how it works. It’s calling set interval, on an interval we’re clearing out
our advertising container we’re just zeroing out the contents of it then it’s get a random
ad from the advertising server, places it with this function here. Placing it just amounts
to it goes out gets the HTML text it needs forward when it’s got it, it sticks it in
the container and binds a click event, but here’s our root issue. Because we’re just
dynamically creating an element and attaching an event to it, at no point are we ever removing
that click handler. So we’re getting hundreds and hundreds of thousands and thousands of
these elements hanging out in the DOM waiting for you to click on them, even though they
are not visible anymore. Because at no point was the execution or was this event ever detached.
So we can see this happening if we just look at Soliloquy, we see this sidebar going off.
Every few seconds it refreshes and puts new ads there. The old ads are still on the page.
Another way of thinking about that over the course of a couple of hours of using Soliloquy
there could be literally millions of David Hasselhoffs hanging out on your page waiting
for you to click on them. {laughter} So my browser crashing the most common source
of these are memory leaks caused by detached DOM elements. You get elements in the DOM
and attached a listener to if you don’t detach the listener when you poll them back out it
can slowly inflate the size of the browser’s memory over time. Which is particularly trouble
some if you have a long lived application like Soliloquy. So these were the 5 JavaScript outlaws that
we tracked down today. Script error is browser opportunities obsfuscation it’s the first
thing you need to track, every time you bring in a third party script you are introducing
risk they can change with your control. Context errors is the most common logical thing passing
the value of this will change. Loading errors because sometimes one of your scripts will
fail to load, causing unique failure conditions in your application. Finally, memory errors
we are leaking DOM or elements are created dynamically. So all of these errors we capture
with a little bit of help from this other project I work on called track.js. If you
are interested in that check that out, this was JavaScript forensic, I was your sheriff
@toddhgardner on Twitter if you use that sort of thing, email me today [email protected]
Thank you very much. {applause}

Add a Comment

Your email address will not be published. Required fields are marked *