tag:blogger.com,1999:blog-56920894819478350062024-03-13T13:42:16.473-07:001Mozillian..2MozilliansRalucahttp://www.blogger.com/profile/07875649704206517173noreply@blogger.comBlogger7125tag:blogger.com,1999:blog-5692089481947835006.post-40171606575726144422014-07-28T17:25:00.000-07:002014-07-28T17:41:50.546-07:00AF (analysis framework) and the task graph story <b>A</b>nalysis <b>F</b>ramework is an application written on top of TaskCluster and provides a way to execute Telemetry MapReduce jobs.<br />
<br />
AF comprises two modules: <br />
<ul>
<li> <a href="https://github.com/raluca-elena/telemetry-analysis-base" target="_blank">telemetry-analysis-base</a>, an example of a MapReduce node</li>
</ul>
<ul>
<li> <a href="https://github.com/raluca-elena/telemetry-analysis" target="_blank">telemetry-analysis</a>, an example of a MapReduce graph that can be posted on TaskCluster </li>
</ul>
<b> </b><br />
<h3>
<b>Telemetry-analysis-base</b></h3>
<br />
This module contains an example of a MapReduce node that can be executed over TaskCluster.<br />
It contains:<br />
<ul>
<li> a custom specification for a Docker image</li>
</ul>
<ul>
<li> a Makefile for creating a Docker image and posting it on registry</li>
</ul>
<ul>
<li> a Vagrantfile useful to developers working on MACOSX </li>
</ul>
<ul>
<li> custom code for map/reduce jobs</li>
</ul>
<ul>
<li> an encryption module used to decrypt STS temporary credentials </li>
</ul>
<br />
<i><b> A task up and running</b><br /> </i><br />
In order to run a task on TaskCluster you need a Docker container, and a custom code to be executed in the container.<br />
A Docker container is a Docker image in execution. To obtain a custom image you can use the Docker specification in the telemetry-analysis-base repository.<br />
As TaskCluster needs a container to run your task in, you need to push your container on TaskCluster registry. The Makefile present in the repository takes care of creating a container and pushing it on the registry.<br />
<br />
Because Telemetry jobs work with data that is not open to the public you need a set of credentials to access the files in S3. The Docker container expects a set of encrypted temporary credentials as an environment variable called CREDENTIALS.<br />
As these credentials are visible in the task description on TaskCluster they are encrypted and base64 encoded. The temporary credentials used are STS Federation Token credentials. These credentials expire in 36 hours and can be obtain only by AWS users that hold a policy to generate them.<br />
After obtaining this credentials they are encrypted with a symmetric key. The symmetric key is encrypted with a public key and sent together as an environment variable to the Docker instance. Inside the Docker container this credentials will be decrypted properly and used to make calls in S3.<br />
<br />
<b><i>Custom code</i></b><br />
<br />
Inside the custom container resides the custom code that will receive as arguments a set of file names in S3. <br />
<br />
<b><i>Mapper</i></b><br />
<br />
After decrypting the credentials the mapper will take the file list and start downloading a batch of files in parallel. As files finish downloading they are stored in a directory called s3/<path is S3> and their names are sent as arguments to the mapperDriver.<br />
MapperDriver will first read the specification of the job from analysis-tools.yml. In the configuration file it is specified if the files need to be decompressed, the mapper function that needs to run and also the language that the mapper is written in.<br />
Next, as the example provided in the repository is in python, the driver spawns another process that executes python-helper-mapper that reads the files, decompresses them, loads the mapper function and sends the decompressed files line by line to the mapper function.<br />
In the mapper function the output is written to result.txt. This file is an artifact for the task ran. <br />
<br />
<b><i>Reducer</i></b><br />
<br />
The reducer task requires an environment variable named INPUT_TASK_IDS specifying all the mapper task ids. Holding the list of all mappers the reducer makes calls to get all the result files from the mappers. As the files finish to download they are stored in a folder called mapperOutput.<br />
The reducerDriver than reads the specification of the job from analysis-tools.yml. The analysis-tools.yml contains the reducer function name and the language that is written in.<br />
In the example provided in the repository the reducer is also written in python so it uses an intermediary module called python-helper-reducer. This module loads the reducer, removes all empty lines from the result files and feeds them to the reducer function.<br />
The output is written to the result file that is an artifact of the reducer task. After writing the result file, the reducer sends an email to the owner of the task. This mail contains a link to the output of the MapReduce job. The email address will be given as an environment variable called OWNER. <br />
<br />
<br />
<br />
<h3>
<b>Telemetry-analysis</b></h3>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiheEhLWMy7AnY93H7uQUMXf_24JofMhpsNP8OAep_aifdHav6TLCW-c3wWhmZZRBOj34JCB5SgSSNyA-4gHSgAzuK4xELCBZyierH1htVPzD7kVZIpAEsgImKvyn87g-yYIxpNrudbT0g/s1600/img.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiheEhLWMy7AnY93H7uQUMXf_24JofMhpsNP8OAep_aifdHav6TLCW-c3wWhmZZRBOj34JCB5SgSSNyA-4gHSgAzuK4xELCBZyierH1htVPzD7kVZIpAEsgImKvyn87g-yYIxpNrudbT0g/s1600/img.jpg" height="400" width="640" /></a></div>
<br />
<br />
This module constructs a taskGraph and posts it to TaskCluster. <br />
At this point a set of credentials is needed to run a task graph: <br />
<ul>
<li>credentials in AWS allowing Federation Token generation. To obtain them you need to specify a policy enabling STS credentials generation.</li>
</ul>
<ul>
<li>public key associated to the private one residing in the running Docker container</li>
</ul>
<ul>
<li>symmetric key used to encrypt the Federation Token credentials</li>
</ul>
<ul>
<li>access to IndexDB</li>
</ul>
<ul>
<li>credentials to TaskCluster (in the future)</li>
</ul>
<br />
<br />
<i> Example call:</i><br />
<br />
<b><i>./AF.js Filter.json "registry.taskcluster.net/aaa" '{"OWNER" : "unicorn@mozilla.com", "BLACK" : "black"}'</i></b><br />
<br />
<br />
AF takes as arguments a Filter.json, a Docker image (registry.taskcluster.net/aaa) and optionally some other arguments that will be passed as environment variables to the Docker container.<br />
AF executes the following:<br />
<br />
<ul>
<li>makes a call for a Federation Token. It encrypts with the public key the credentials and provides base64 encrypted credentials as CREDENTIALS environment variable to the Docker container</li>
</ul>
<ul>
<li>using Filter.json AF queries indexDB to get the specific file names and file sizes</li>
</ul>
<ul>
<li>creates skeletons for mapper tasks and adds load to them (file names from indexDB)</li>
</ul>
<ul>
<li>pushes taskDefinition to graph Skeleton</li>
</ul>
<ul>
<li>creates reducer task and gives it as dependencies the labels of dependent tasks</li>
</ul>
<ul>
<li>posts the graph</li>
</ul>
<ul>
<li>gets the graph definition, prints the graph definition and a link to a simple monitor page</li>
</ul>
<ul>
<li>as the graph finises execution, the page will contain the links to the result page </li>
</ul>
<br />
<b>Last but not least</b><br />
<br />
<br />
Analysis Framework is a really interesting/fun project. It can be easily extended or reused, it is designed as an example that can be customized and has some documentation too. :p<br />
<br />
<br />
<br />
<br />Ralucahttp://www.blogger.com/profile/07875649704206517173noreply@blogger.com0tag:blogger.com,1999:blog-5692089481947835006.post-17576736261270845142014-06-29T22:43:00.000-07:002014-06-29T22:44:24.612-07:00Like all magnificent things, it's very simple.<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="font-size: small;">While working on the Dashboard I learned about the analysis workflow. This is a diagram stolen from <a href="http://jonasfj.dk/blog/2013/11/telemetry-rebooted-analysis-future/#comments" target="_blank">Jonas</a>:</span></span><br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_zkQdAYvR59j3kLhGwY2wJdgpa-AH2mvEILHdtesZtQLYYGLdXXerrTg4zZZd_FEw-jm1UFlUnVbXojfNfRaynF5lENh1u502mmrTyeZZW0iP4jki0KxQX3wJmYBwVwFTO10D-38fpmA/s1600/telemetry-dashboard-flow.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_zkQdAYvR59j3kLhGwY2wJdgpa-AH2mvEILHdtesZtQLYYGLdXXerrTg4zZZd_FEw-jm1UFlUnVbXojfNfRaynF5lENh1u502mmrTyeZZW0iP4jki0KxQX3wJmYBwVwFTO10D-38fpmA/s1600/telemetry-dashboard-flow.png" height="400" width="337" /></a></span></span></b></div>
<b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><br /></span></span></b>
<b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><br /></span></span></b>
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="font-size: small;">Two weeks ago I started working on Telemetry Analysis Framework.</span></span><br />
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="font-size: small;">We are simplifying the MapReduce workflow to be as flexible yet as easy to use and debug as possible. Jonas has been developing <a href="http://docs.taskcluster.net/" target="_blank">TaskCluster</a> for a while and he came up with the idea of porting the analysis to it.</span></span><br />
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span>
<br />
<h3>
<span style="font-size: large;"><b><span style="font-weight: normal;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;">What is TaskCluster</span></span></b></span></h3>
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="font-size: small;"><span style="background-color: white; box-sizing: border-box; color: #333333; font-style: normal; font-variant: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">TaskCluster is</span><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><span class="Apple-converted-space"> </span></span></span><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">a set of components that manages task queuing, scheduling, execution and provisioning of resources. It is designed to run automated builds and tests at Mozilla. You can imagine it like the diagram below:</span></span></b></span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2QYZOQaFLXdaz6JBA8J3WhOu4um6BFN03kR0ORCcD50FlbyoX6zN3Keq_6aGqozfs-UaZe9M-Nuinabqa_zXj1Myih3M768z_ms9-KCfw6Dhzhgs8zR2wxHgG1H3griMi8VhIuLPqZGc/s1600/2014-06-29+(1).png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2QYZOQaFLXdaz6JBA8J3WhOu4um6BFN03kR0ORCcD50FlbyoX6zN3Keq_6aGqozfs-UaZe9M-Nuinabqa_zXj1Myih3M768z_ms9-KCfw6Dhzhgs8zR2wxHgG1H3griMi8VhIuLPqZGc/s1600/2014-06-29+(1).png" height="320" width="400" /></a></span></span></b></div>
<b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span></span></span></b>
<br />
<h3>
<span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"> </span></b></span></span></h3>
<h3>
<span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Task</span></b><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">C</span><b><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">luster for our MapReduce Analysis</span></b></span></span></h3>
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">When you </span><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">talk about</span><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"> MapReduce </span></span></b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">it is usually referring to a directed graph workflow like the one below.</span> Taskcluster provides task graphs that can easily be used as a MapReduce workflow.</span></span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJPsI3Xgr0Kl8052KUHGltylIcG7Y-WYk7iAZK-22WI__ZUeOYk0Tfd6uOTCDVFUU8P3eJkfClhl2cJhkZIpoL5duVAu7BIqWmj4SJuymyAIsZBcbK0OsAba3Nh-rsrZQKLK5mmSbJpHs/s1600/Sketch290103323.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJPsI3Xgr0Kl8052KUHGltylIcG7Y-WYk7iAZK-22WI__ZUeOYk0Tfd6uOTCDVFUU8P3eJkfClhl2cJhkZIpoL5duVAu7BIqWmj4SJuymyAIsZBcbK0OsAba3Nh-rsrZQKLK5mmSbJpHs/s1600/Sketch290103323.png" height="320" width="400" /></a></span></span></b><b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"> </span></b></span></span></b><b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"> </span></b></span></span></b></div>
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">As mentioned above we want this framework to be:</span></span></b></span><br />
<ul>
<li><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">simple to use</span></span></b></span></li>
<li><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">flexible</span></span></b></span></li>
<li><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">easy to debug</span></span></b></span></li>
</ul>
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="font-size: small;"></span></b></span><br />
<br />
<span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Simple to use </span></b></span></span><br />
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Simplicity in this case means the programmer has to specify as </span></span></b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">little</span></span><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"> as possible:</span></span></b></span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgC8_HIf7cSFIM9azhpMHBYyQraWlNOzJJ9RFeCcZkMZ7awbdLUYFHx2gt2YTFCgLDyYjraETJyx8cs29RV_ZVBITeznk4RmtNa8FPrrQBidqUsz_qYaaqgugP6nzw4sYL6g_MHBptV3Fk/s1600/Sketch290124829.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgC8_HIf7cSFIM9azhpMHBYyQraWlNOzJJ9RFeCcZkMZ7awbdLUYFHx2gt2YTFCgLDyYjraETJyx8cs29RV_ZVBITeznk4RmtNa8FPrrQBidqUsz_qYaaqgugP6nzw4sYL6g_MHBptV3Fk/s1600/Sketch290124829.png" height="320" width="400" /></a></span></span></b><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"> </span></span></b></span></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<b><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Docker Image</span></span></span></b></div>
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">For the purpose of this application you can consider a Docker image as a lightweight virtual machine.<b> </b>In it you store the setup for the analysis: needed utilities and custom code.</span></span></b></span><br />
<br />
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Because we know that starting with something new can be annoying we provide a base image where you only need to write your custom code. Also we provide dead easy documentation for <b>each</b> step.<b> </b>More information about Docker can be found <a href="http://docs.docker.com/" target="_blank">here</a>.</span></span></b></span><br />
<h3>
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"> </span></span></b></span></h3>
<h3>
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Filter.json</span></span></span><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"> </span></span></span></h3>
<h3>
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Filter.json is the file given so we can extract from S3 bucket the files needed for the </span></span></b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">analysis</span></span><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">.</span></span></b></span></h3>
<br />
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">By providing </span></span></b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">these </span></span><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">two elements the framework will proceed as follows:</span></span></b></span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg3fIRqNsJLi-HRswOdfpeFubzkhj4NuEqheDlFZDlYWmrxygTfmTSDS6kJXAlys1PeLIxeRgo7ZWF-iDLMih9bWe9qC2aMm1PQcJo6_8QR0c6ReUpIO0w0Sp64_wikpJkPULSTEIm7tGc/s1600/Sketch275155456.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg3fIRqNsJLi-HRswOdfpeFubzkhj4NuEqheDlFZDlYWmrxygTfmTSDS6kJXAlys1PeLIxeRgo7ZWF-iDLMih9bWe9qC2aMm1PQcJo6_8QR0c6ReUpIO0w0Sp64_wikpJkPULSTEIm7tGc/s1600/Sketch275155456.png" height="320" width="400" /></a></span></span></b></div>
<br />
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">The Analysis Framework will parse the Filter.json, make a request to the index database, split the response in sublists of files and start a map task for <b>each batch of files</b>.</span></span></b></span><br />
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span></span></b></span>
<span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">A map </span></span><b><span style="font-size: small;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">task would look<b> </b>as follows:</span></span></b></span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnZDV2ZMecPm2PbFBxT_FMFcz7URftYuct1UBYbIQlJ4i-wPljuOx6Tph8BtRqFLiGXHoNPaV6LfB10zDbNJAbRVsf7n6m8JR9ce7GDN8ZAx_9LfSM18r6rlS8b7bxdBXTewj3sbw27Nk/s1600/Screen+Shot+2014-06-29+at+13.35.42.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnZDV2ZMecPm2PbFBxT_FMFcz7URftYuct1UBYbIQlJ4i-wPljuOx6Tph8BtRqFLiGXHoNPaV6LfB10zDbNJAbRVsf7n6m8JR9ce7GDN8ZAx_9LfSM18r6rlS8b7bxdBXTewj3sbw27Nk/s1600/Screen+Shot+2014-06-29+at+13.35.42.png" height="584" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"> <b> </b></span></span></span></b><br />
<b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"> </span></span></span></b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">For a mapper task we would need to specify the Docker image (line 25), a command (line 27) and the files that need to be processed (lines 28 and 29).</span></span></span><br />
<span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">The output of this task will be an artifact /bin/mapper/result.txt and will be uploaded to an intermediate bucket on Amazon.</span></span></span><br />
<br />
<h3>
<span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Flexible</span></span></span></h3>
<span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">We call this framework flexible because you can customize all the layers provided.</span></span></span><br />
<span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">The framework comes with a downloader utility, several mapper helpers (python, javascript) that can decompress the files and feed the result to the loaded custom map function.</span></span></span><br />
<span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">If you would like a different Docker image you can customize that too. The framework includes a default image and also the Docker file out of which you can build the image. By modifying that file you can easily get another Docker image that suits your needs.</span></span></span><br />
<br />
<span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">If you are on MacOS you probably need some way to work with Docker. By installing <a href="http://www.vagrantup.com/" target="_blank">Vagrant</a> and adding the Vagrantfile to your directory you can work easily with Docker on your machine.</span></span></span><br />
<br />
<h3>
<span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Easy to debug</span></span></span></h3>
<span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Until now the analysis that we performed had close to zero logging. This had to change so we could get a robust and easy way to debug the workflow. With the approach we are trying to develop right now we have logging at each step. The developer has the option to see how many rows were processed, what were the files downloaded with success, what were the ones decompressed successfully and if an error occurred. With this information some retry policies could be implemented. </span></span></span><br />
<br />
<span style="font-size: small;"><span style="font-size: large;"><b><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Last but not least</span></span></b></span><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"> </span></span></span><br />
<span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">I will be blogging about this next week and also provide you with the link to the official repo. Right now a lot of changes happen in my playground :) .. </span></span></span><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"></span></span></span><b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span></span></span></b>
<b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span></span></span></b>
<b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span></span></span></b>
<b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="background-color: white; color: #333333; display: inline ! important; float: none; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 20px; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br /></span></span></span></b>
<br />
<b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><br /></span></span></b>
<b><span style="font-size: small;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><br /></span></span></b>
Ralucahttp://www.blogger.com/profile/07875649704206517173noreply@blogger.com0tag:blogger.com,1999:blog-5692089481947835006.post-36799977178024809262014-06-10T10:28:00.000-07:002014-06-10T10:28:46.403-07:00“Alice:How long is forever? White Rabbit:Sometimes, just one second.” <span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Last week I finally pushed the new dashboard to production.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The problem with bugs is that ...the later you find them the harder they bite your ass. I won't talk about bugs because you don't talk bad about the dead :p</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span style="background-color: white;"><b style="color: #242a2c; line-height: 27px;">Telemetry Dashboards</b><span style="color: #242a2c; line-height: 27px;"> measure how Firefox </span><span style="color: #242a2c; line-height: 27px;">behaves on desktop and mobile in real world use.</span></span><span style="color: #242a2c; line-height: 27px;">This application provides the aggregate view of the </span><span style="color: #242a2c; line-height: 27px;">data collected.</span></span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">One can hit <a href="http://telemetry.mozilla.org/">http://telemetry.mozilla.org/</a> in three ways:</span><br />
<br />
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">request for default page (no cookie set)</span></li>
</ul>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">request with cookie set (cookie expiration 3 days)</span></li>
</ul>
<ul>
<li><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">request for specific state (aka url encoding a particular state)</span></li>
</ul>
<br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The page itself has two states:</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b>Single filter </b>(when you look for a "one at a time version plot") you get histogram/table plots and summary details.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhydR8qK7jKSZ0q93lfs35YHHTCu9hXuOBC3iKOmF23lt1XM8uecpKpAS15FkMP-iT2V0pRYcAUXycBVN9pT2Xydlixjd9MAJcOFlJminiGCuEsI_nlwE5GiLH4u2d7_zEHwDVN16MpLyM/s1600/singleMode.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhydR8qK7jKSZ0q93lfs35YHHTCu9hXuOBC3iKOmF23lt1XM8uecpKpAS15FkMP-iT2V0pRYcAUXycBVN9pT2Xydlixjd9MAJcOFlJminiGCuEsI_nlwE5GiLH4u2d7_zEHwDVN16MpLyM/s1600/singleMode.png" height="368" width="640" /></span></a></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Several filters</span></b><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEilIkrUu0jOSCmZd7Yfa5SB5xO1GttOIVl2Y77WwMdjgQBuMBAyaJyrSrp0yUrtRe_n7zc9wIhHtPVZrTmP7D9g3vU3b-qgrxPnG1vALHWIuSMOEPnN-Js2cfMFu6s6AyU7zxbz-o9iR3M/s1600/MultipleVersions.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEilIkrUu0jOSCmZd7Yfa5SB5xO1GttOIVl2Y77WwMdjgQBuMBAyaJyrSrp0yUrtRe_n7zc9wIhHtPVZrTmP7D9g3vU3b-qgrxPnG1vALHWIuSMOEPnN-Js2cfMFu6s6AyU7zxbz-o9iR3M/s1600/MultipleVersions.png" height="324" width="640" /></span></a></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">By adding multiple filters on your plot you switch from one mode to another. If you want to get back into single mode state you just have to remove all the filters you added.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Locked/Unlocked state</span></b><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">There are two ways you can use the filters:</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Locked State</span></b><br />
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></b>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">You want all the measures to be in sync with the first filter. Whatever change made in your first filter will be propagated to the ones below. If the ones below don't have data for this particular measure they will be set to a default measure.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirjgkOGr23c7bHbD-YYDTUmfQ68053pwnlk_eDpQeqKUBIXc7SmkR5GgRJWg1YYDzTbZTdPqb0UUY2L4eaFxymmBBOuJeFUeBQANh36Dll3TptaYur9UF8KBa7l3CkrGeApcEQeXgc-as/s1600/default.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirjgkOGr23c7bHbD-YYDTUmfQ68053pwnlk_eDpQeqKUBIXc7SmkR5GgRJWg1YYDzTbZTdPqb0UUY2L4eaFxymmBBOuJeFUeBQANh36Dll3TptaYur9UF8KBa7l3CkrGeApcEQeXgc-as/s1600/default.png" height="328" width="640" /></span></a></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Unlocked State</span></b><br />
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></b>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Use the unlock button to compare between different measures. </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyjMJZtqFoVRn7lqwyl8iQQldP1pwIfe7ADk2TxT0Tbh50zCbA_v4_Vivg4POq-afuTph7fhIwXWnTg6_sKz0kiezPnXlPpMJ5XLBwuME125T-Wuqo1OPYcIU_HcM5S8SJHdlK7gQ1zF4/s1600/NOTSYNC.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyjMJZtqFoVRn7lqwyl8iQQldP1pwIfe7ADk2TxT0Tbh50zCbA_v4_Vivg4POq-afuTph7fhIwXWnTg6_sKz0kiezPnXlPpMJ5XLBwuME125T-Wuqo1OPYcIU_HcM5S8SJHdlK7gQ1zF4/s1600/NOTSYNC.png" height="363" width="640" /></span></a></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Legend</span></b><br />
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></b>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The filters can get very long. In some cases it doesn't make sense to show the full state because we only have one measure across multiple versions in the locked state. That's why we show the title of the measure and the details for it and simplify the legend by making it show only the versions.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgH_vnbcxDJfu73Eu8PAeombSZ2oopYdbTyW37f4dwz0yRJpbtbrNa6CxZqY3oC3DFt2WaSFk5Bxk7I4nhmrGX1BvoDbBFDAQXWQeiySFJLWGuPLG2gM9gmtSNFP0QKYxFeadUF1bWSUP8/s1600/legend.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgH_vnbcxDJfu73Eu8PAeombSZ2oopYdbTyW37f4dwz0yRJpbtbrNa6CxZqY3oC3DFt2WaSFk5Bxk7I4nhmrGX1BvoDbBFDAQXWQeiySFJLWGuPLG2gM9gmtSNFP0QKYxFeadUF1bWSUP8/s1600/legend.png" height="328" width="640" /></span></a></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In the case of mixed filters (aka unlocked state) we show only the part of the filter that fits one line, for more information you can look in the tooltip.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHT-BJYTvctrUMW3eXYJLuddTYAyUr7JQ0pli0_-8y2MsZ2tNj0c-LTV2QxGKOwkeKKBc9DjL4PO9agCUm_c0ElXsPMp46TyvD_RLkgjxb-Sm1OlZTxV8PAnl8vRWgba-jMA9j4bVKMT8/s1600/tooltip1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHT-BJYTvctrUMW3eXYJLuddTYAyUr7JQ0pli0_-8y2MsZ2tNj0c-LTV2QxGKOwkeKKBc9DjL4PO9agCUm_c0ElXsPMp46TyvD_RLkgjxb-Sm1OlZTxV8PAnl8vRWgba-jMA9j4bVKMT8/s1600/tooltip1.png" height="237" width="400" /></span></a></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Caching</span></b><br />
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></b>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">When you add another filter we instantiate a new selector but we don't make a request. Requests are done only if we don't have that data in the cache.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Url and TinyUrl</span></b><br />
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></b>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">All the page state: filters, lock/unlock, selector state (percentiles), table/histogram are preserved in the url. This way the state gets saved for a return with state or cookie session.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmCyWBZnbEAhhr4Q1zcm6VsIrh7Vzj6liNAT_bvd_r39UURwjV2PdcgidLbLYM4tsHo64mSrdXw-DYW5QaWSnNYGB4OhIUPfVN-_OwdnUtp47mXi0EC6ZpntX3Y9OwDyUabc7eQ21HQKA/s1600/tinyUrl.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmCyWBZnbEAhhr4Q1zcm6VsIrh7Vzj6liNAT_bvd_r39UURwjV2PdcgidLbLYM4tsHo64mSrdXw-DYW5QaWSnNYGB4OhIUPfVN-_OwdnUtp47mXi0EC6ZpntX3Y9OwDyUabc7eQ21HQKA/s1600/tinyUrl.png" height="328" width="640" /></span></a></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><b><i>The end</i></b>... :)</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDH-tlkEqVHLB_YcoG8tv2MUSKUkW95QzbSIxBOgMpMHCoTiSpiKfQI4upijCjdu9O_kmHwtRRwK0otbbuHIIPcNHev8EB3CMInaVkGGlgK4dae6Joto6i8ZOJzZHmxnFeifMJ-OJhfC8/s1600/paradise.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDH-tlkEqVHLB_YcoG8tv2MUSKUkW95QzbSIxBOgMpMHCoTiSpiKfQI4upijCjdu9O_kmHwtRRwK0otbbuHIIPcNHev8EB3CMInaVkGGlgK4dae6Joto6i8ZOJzZHmxnFeifMJ-OJhfC8/s1600/paradise.png" height="353" width="640" /></span></a></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
Ralucahttp://www.blogger.com/profile/07875649704206517173noreply@blogger.com1tag:blogger.com,1999:blog-5692089481947835006.post-1548979599495787202014-05-27T23:50:00.000-07:002014-05-28T10:01:22.416-07:00It's the little things that matter<h3>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Little things called bugs...</span></h3>
<h4>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Silent crash on specific data</span></h4>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br />
I started the past week by investigating a bug. It can be reproduced on the current dashboard:</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br />
</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6b0LAqjNx5Jx1mufa5d3GWQj8DvWQfimKNOZ4RZwi9xufyEiYp4_aL60u34Ugb-rNJawWEjstIX8vEzLUb1QZc1i8yG5zjfWBKssw6D0FmVFS4YxkLscaeSDGp68cUrDaM9Ub3bTTsJw/s1600/Screen+Shot+2014-05-27+at+17.51.28.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6b0LAqjNx5Jx1mufa5d3GWQj8DvWQfimKNOZ4RZwi9xufyEiYp4_aL60u34Ugb-rNJawWEjstIX8vEzLUb1QZc1i8yG5zjfWBKssw6D0FmVFS4YxkLscaeSDGp68cUrDaM9Ub3bTTsJw/s1600/Screen+Shot+2014-05-27+at+17.51.28.png" height="303" width="320" /></span></a></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br />
<br />
<br />
Updating to the latest versions of d3 and nvd3 didn't do the trick so I spent some time looking at the data and the callstack. This plot uses a library called nvd3 that is built on top of d3. The plot calls the voronoi method (v<span style="background-color: white; color: #333333; line-height: 20.399999618530273px;">oronoi layouts). </span><span style="background-color: white; color: #333333; line-height: 20.399999618530273px;">This</span><span style="box-sizing: border-box; color: #333333; line-height: 20.399999618530273px;"><span style="background-color: white;"> method </span></span><span style="background-color: white; color: #333333; line-height: 20.399999618530273px;">returns an array of polygons, one for each input vertex in the specified</span><span style="background-color: white; color: #333333; line-height: 20.399999618530273px;"> </span><span style="box-sizing: border-box; color: #333333; line-height: 20.399999618530273px;">data</span><span style="background-color: white; color: #333333; line-height: 20.399999618530273px;"> </span><span style="background-color: white; color: #333333; line-height: 20.399999618530273px;">array. If any vertices are coincident or have NaN positions,</span><span style="background-color: white; color: #333333; line-height: 20.399999618530273px;"> </span><span style="box-sizing: border-box; color: #333333; line-height: 20.399999618530273px;">the behavior of this method is undefined</span><span style="background-color: white; color: #333333; line-height: 20.399999618530273px;">: most likely, invalid polygons will be returned. </span></span><br />
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 20.399999618530273px; margin-bottom: 15px; margin-top: 15px;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In our case we've got points that are coincident so..we get undefined behavior. The fix is adding some random tiny noise to the data. </span></div>
<h4 style="background-color: white; box-sizing: border-box; color: #333333; line-height: 20.399999618530273px; margin-bottom: 15px; margin-top: 15px;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> Tooltip undesired behavior</span></h4>
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 20.399999618530273px; margin-bottom: 15px; margin-top: 15px;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> Sometimes, on the current dashboard, the tooltip doesn't show up. After testing for a while I realized that if you refresh the page (and get an empty svg) it works, but if you don't the tooltip might not show up. Emptying the svg resolves this problem. </span></div>
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 20.399999618530273px; margin-bottom: 15px; margin-top: 15px;">
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Memoization</span></b></div>
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 20.399999618530273px; margin-bottom: 15px; margin-top: 15px;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> Until now we were preprocessing the data at any filter entry, no matter if we had duplicated data or not (x_x). It was about time to add some cache support. I added some caching in both dashboard.js and in telemetry.js. In dashboard.js we cache all the data that we prepared for plotting and in telemetry.js we cache all the requests made (patch <a href="https://github.com/raluca-elena/telemetry-dashboard/commit/09b3c16fddc104dbebf41c86c7c66fd5fa7e968a" target="_blank">here</a>). </span></div>
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 20.399999618530273px; margin-bottom: 15px; margin-top: 15px;">
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">TinyUrl</span></b></div>
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 20.399999618530273px; margin-bottom: 15px; margin-top: 15px;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Because we sync with all the events on the page the url may get really long so "tiny url" is a nice to have feature. I looked at the free options and picked <a href="https://bitly.com/" target="_blank">bitly</a> for it's jsonp call support. Because of the same-origin policy you can't make normal calls to a different domain so you either implement CORS or stick to JSONP.</span></div>
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 20.399999618530273px; margin-bottom: 15px; margin-top: 15px;">
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">New legend</span></b></div>
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 20.399999618530273px; margin-bottom: 15px; margin-top: 15px;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">A nice clean legend for several series was on my list last week. I tried to implement it with d3 and didn't get to the result I wanted so I switched to JQuery and added some d3 formatting.</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6Pp9BRQHbe5K0Uu1Pqs4PHbEFFaNfMhlu4UuulyFQrNdJNGNZ0rTmhRC-hJ8SH4O5TKczK-azP80jkhhpZM9Oydt9HE4OZ29RoRpv8eQ7K_yvbP7p5_XnaDevwgI2LeG0Ar1Z3GRwYOA/s1600/Legend.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6Pp9BRQHbe5K0Uu1Pqs4PHbEFFaNfMhlu4UuulyFQrNdJNGNZ0rTmhRC-hJ8SH4O5TKczK-azP80jkhhpZM9Oydt9HE4OZ29RoRpv8eQ7K_yvbP7p5_XnaDevwgI2LeG0Ar1Z3GRwYOA/s1600/Legend.png" height="201" width="320" /></span></a></div>
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 20.399999618530273px; margin-bottom: 15px; margin-top: 15px;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 20.399999618530273px; margin-bottom: 15px; margin-top: 15px;">
<h4>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Last but not least... </span></h4>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Fresh air (or steam) comes with fresh ideas. Check out <a href="http://www.nps.gov/yell/index.htm" target="_blank">Yellowstone</a> :)</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibGktbPMJzQLwI0jOg6HHn47KerUGzoRA4CHvCOclRSXE1ImmVfIg14lJgw3wIK26QALgYmWCafb2PBT1g485-kYTwRlAcMB347PIxsIt1uF7_ZTcsB0ZZC3qvhfkA7JIHk2vlUB6Y0bw/s1600/DSC05488.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibGktbPMJzQLwI0jOg6HHn47KerUGzoRA4CHvCOclRSXE1ImmVfIg14lJgw3wIK26QALgYmWCafb2PBT1g485-kYTwRlAcMB347PIxsIt1uF7_ZTcsB0ZZC3qvhfkA7JIHk2vlUB6Y0bw/s1600/DSC05488.JPG" height="213" width="320" /></a></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 20.399999618530273px; margin-bottom: 15px; margin-top: 15px;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 20.399999618530273px; margin-bottom: 15px; margin-top: 15px;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 20.399999618530273px; margin-bottom: 15px; margin-top: 15px;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> </span></div>
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 20.399999618530273px; margin-bottom: 15px; margin-top: 15px;">
<span style="font-family: Consolas, Liberation Mono, Courier, monospace;"><span style="font-size: 12px;"><br /></span></span></div>
Ralucahttp://www.blogger.com/profile/07875649704206517173noreply@blogger.com0tag:blogger.com,1999:blog-5692089481947835006.post-88934274813377997752014-05-20T11:33:00.000-07:002014-05-21T18:19:23.157-07:00A step backward, after making a wrong turn, is a step in the right direction..<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The version* feature</span></b><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">As a nice to have feature we could aggregate all the data in a single series for all nightly/aurora/etc.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">I spent one day on this matter and ..it can be done ...but let me explain why not:</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">At this point when we make a request for a specific versions/series/etc we get three jsons:</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">* filter-tree.json</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">* histograms.json</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">* [measure]-[by-build-date/-by-submission-date].json</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In histograms.json we get all the measures while in filter-tree.json a tree of filters. </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">After traversing the filter tree and selecting the filter ids we take the list of ids and look in [measure]-[by-build-date/-by-submission-date].json at each entry. Whenever finding a corresponding id we append the data that afterwords is plotted.</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Imagine doing that for all versions add a large quantity of data. Remember that the processing is done on the client side (maybe on a slow device?)... it might end up in flames :(...</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">End of version* story for now...(it could always be done on server side)</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Several series. Synchronize with top filter</span></b></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">If you want to preserve all your selections for another version you can do that by synchronizing with the first filter and the selection will get propagated on all other filters. </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Because I added "sync with hash" now you can land on the dashboard in two ways:</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> * by specific state (example:/index.html#nightly/32/CYCLE_COLLECTOR&nightly/25/CYCLE_COLLECTOR) </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In this case the filters are not synchronized. If you want to synchronize them you just click the lock button.</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">* by a default landing</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">On the default page the sync with first is enabled. If you want to disable the sync you just push the lock button once.</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDL31IENzq7V_5EATUjObnenDGcOm6oT3U51PMEgqgBiI7HjIrpUI4tKa8oV5tLgQ3Ta-hSG_fUEt4EP7FKTfca42DXY6ASJFTE6q66C7qqnpJuFMyBGZ-orarcc-sa7h9HVj_gjUimmA/s1600/lock.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDL31IENzq7V_5EATUjObnenDGcOm6oT3U51PMEgqgBiI7HjIrpUI4tKa8oV5tLgQ3Ta-hSG_fUEt4EP7FKTfca42DXY6ASJFTE6q66C7qqnpJuFMyBGZ-orarcc-sa7h9HVj_gjUimmA/s1600/lock.png" height="282" width="640" /></span></a></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
</div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Sync with state</span></b></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In this version you can preserve your options for another time by saving the url from the dashboard and plotting it in a browser afterwords. This weeks version will come with a refactored and detailed one using some jquery magic. It will get some tinyUrl option too so it can be easily dropped on irc.</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">From <a href="http://getbootstrap.com/">Boo<span id="goog_291830644"></span><span id="goog_291830645"></span>tstrap</a> to <a href="http://ivaynberg.github.io/select2/">Select2</a></span></b></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">By adding multiple versions at some point my dashboard started dying..I began profiling and I got lucky.</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">In the previous version we used a library called Bootstrap, we had a default implementation for a simple selector that was extended with some shiny bootstrap magic. All was awesome till we started adding several selectors ... </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Multiple things might be the cause:</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> * big overhead for computing and aggregating the data</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> * d3/nvd3 lib </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> * selectors</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> * bad synchronization </span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"> * me writing horrible and unoptimized code</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">The last three were a quick fix :D</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">After switching to a different library, modifying the selector and resolving some bugs I ended up with this comparison:</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiV0tW_rhJQYYRMGijgpYkc2_vhJMxvOVm7oGQ_032ar5tr7abwhe5R2m4FSgyES8LnNSLdOt3DefcjfD7ZWi5H87inBqHZdsLuUdORlWpZZbYSh1raxNkiGDkT2wLcZi2xqeW2UNpam6Y/s1600/AddVersionComparison.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiV0tW_rhJQYYRMGijgpYkc2_vhJMxvOVm7oGQ_032ar5tr7abwhe5R2m4FSgyES8LnNSLdOt3DefcjfD7ZWi5H87inBqHZdsLuUdORlWpZZbYSh1raxNkiGDkT2wLcZi2xqeW2UNpam6Y/s1600/AddVersionComparison.jpg" height="400" width="640" /></span></a></div>
<div class="separator" style="clear: both; text-align: center;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">If you take <span style="text-align: center;">the same mix of series on load and look at the time spent in the selector you get interesting findings:</span></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; text-align: center;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoDrErIVG4OtADG_g5ao3kukydZA36sIzR44T2VG8v-c0TpQsJya64gnhW30k5IIprR_sS7mWbHD7iSxDeQ7Y_d-6HCLlm08A-0-jJFmjtfxhg45_g5cCCICPqGB6P4tt_TyGpCUrIkz4/s1600/SelectorComparison.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoDrErIVG4OtADG_g5ao3kukydZA36sIzR44T2VG8v-c0TpQsJya64gnhW30k5IIprR_sS7mWbHD7iSxDeQ7Y_d-6HCLlm08A-0-jJFmjtfxhg45_g5cCCICPqGB6P4tt_TyGpCUrIkz4/s1600/SelectorComparison.jpg" height="400" width="640" /></span></a></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif; text-align: center;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
</div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Select2 FTW!</span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Temporary fixes and D3</span></b></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">On the production dashboard the plot comes with a very nice legend that turns unusable when you add several versions. On this weeks version I had to fix that. I removed the legend and put it aside in a scrollable svg. It turns out that even so it is cluttered. I removed the information that doesn't make sense for multiple versions: Summary & Histogram. The functionality is still there but is set visible only for one filter or land on the default page. </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<b><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">This week</span></b></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Today I had an interesting discussion with one of the users and it turns out that the Summary and Histogram would be a nice to have for multiple versions too. A zoom in the histogram would be nice :D. </span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">I think I figured out why D3 crashes when it gets some particular type of data..(production bug)</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgP-BkycmT10M1E8TesMX3ulEVJ-9SXjnJ8IJZzX0d-8nP3_53j-2IGEEZz7vAD3_hvK0IMMQPBg0s9KbXfx9ctDklPYczQAX2vfjz6P3xZgfP_zg3Ynfd8pDy2FQuVkQIqVe-Wk4vlOPU/s1600/Screen+Shot+2014-05-20+at+9.35.33.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgP-BkycmT10M1E8TesMX3ulEVJ-9SXjnJ8IJZzX0d-8nP3_53j-2IGEEZz7vAD3_hvK0IMMQPBg0s9KbXfx9ctDklPYczQAX2vfjz6P3xZgfP_zg3Ynfd8pDy2FQuVkQIqVe-Wk4vlOPU/s1600/Screen+Shot+2014-05-20+at+9.35.33.png" height="353" width="400" /></span></a></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 14.571428298950195px; margin-bottom: 15px; margin-top: 15px;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 14.571428298950195px; margin-bottom: 15px; margin-top: 15px;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span style="background-color: transparent;">I went for a quick fix, but that is not a long term solution. Still looking for some other options. My guess, this bug triggers the tooltip undesired behavior too. </span><span style="background-color: transparent;">Getting at the bottom of this might fix more than one problem.</span></span></div>
</div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span style="background-color: #3879d9; color: white; line-height: 17px; white-space: nowrap;"><br /></span>
</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span style="background-color: #3879d9; color: white; line-height: 17px; white-space: nowrap;"><br /></span><span style="background-color: #3879d9; color: white; line-height: 17px; white-space: nowrap;"><br /></span><span style="background-color: #3879d9; color: white; line-height: 17px; white-space: nowrap;"><br /></span><span style="background-color: #3879d9; color: white; font-size: 11px; line-height: 17px; white-space: nowrap;"><br /></span></span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">
</span><br />
<br />Ralucahttp://www.blogger.com/profile/07875649704206517173noreply@blogger.com0tag:blogger.com,1999:blog-5692089481947835006.post-52664887483624319982014-05-12T14:35:00.000-07:002014-05-12T14:35:01.396-07:00The Good, the Bad and the Ugly <span style="font-family: Times, Times New Roman, serif;">I just finished my second week as "the intern" that works on Telemetry Dashboard.</span><br />
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<span style="font-family: Times, Times New Roman, serif;">After my first blog post I got some feedback that made me switch<i> </i>between<i> </i>tasks<i>.</i></span><br />
<span style="font-family: Times, Times New Roman, serif;">Although <a href="http://dygraphs.com/">dygraphs</a> sounds like an option and <a href="http://d3js.org/">d3</a> can be a pain in the ass I switched back to </span>the latter<span style="font-family: Times, 'Times New Roman', serif;"> for flexibility reasons (d3 can be pretty powerful you know:p).</span><br />
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<span style="font-family: Times, Times New Roman, serif;">I spent a day or two trying to figure out a way to make the bootstrap selectors for versions/measure take multiple versions. The selector was written "one version/measure at a time" and because all the filters were tightly linked together the code got uglier and uglier...</span><br />
<span style="font-family: Times, Times New Roman, serif;">till my eyes almost started bleeding so I said...enough!...let's find a fast workaround...and maybe on the way we get things done even better :p </span><br />
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<span style="font-family: Times, Times New Roman, serif;">The "right in the face" option was to add the same kind of selector for each new version dynamically. </span><br />
<span style="font-family: Times, Times New Roman, serif;">I needed some new elements and functionality: add/remove buttons.</span><br />
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_unqgJgaeyXlGuimgVxe9ylMPQPUb-2L2ndHYBMUcNt5OLKj-VjXwDmEw2VROxBr-jiAIv5AbKyeMLcK37ZykIf6jUwMPfzRCBNF4dzoxiJ6FyGycBQLjH2ggZ3UMb6scfBk9ngzmDH8/s1600/img1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Times, Times New Roman, serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_unqgJgaeyXlGuimgVxe9ylMPQPUb-2L2ndHYBMUcNt5OLKj-VjXwDmEw2VROxBr-jiAIv5AbKyeMLcK37ZykIf6jUwMPfzRCBNF4dzoxiJ6FyGycBQLjH2ggZ3UMb6scfBk9ngzmDH8/s1600/img1.png" height="180" width="400" /></span></a></div>
<div class="separator" style="clear: both; text-align: center;">
<span style="font-family: Times, Times New Roman, serif;"><br /></span></div>
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<span style="font-family: Times, Times New Roman, serif;">This way we comply with the feature request: "add multiple versions on the same plot" and also we can add different measures for each specific version (if that might be the case).</span><br />
<span style="font-family: Times, Times New Roman, serif;">Next step was to add a new multiple selector so that we can automatically select the submission/percentile/etc over several versions:</span><br />
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZu6LOTGgajg0UI1CJ339D-Jqy_mA3xOAXTVXDZ5QKwXJLJqEnIvSerurtGBNflHM7GikeTF16_nBl9oxsB6Q_nuAO77nF_cYCZdaUHKaf5EKBELioTpCSwo0HsUGtPzFMC_Tdx4sfmRw/s1600/img2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Times, Times New Roman, serif;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZu6LOTGgajg0UI1CJ339D-Jqy_mA3xOAXTVXDZ5QKwXJLJqEnIvSerurtGBNflHM7GikeTF16_nBl9oxsB6Q_nuAO77nF_cYCZdaUHKaf5EKBELioTpCSwo0HsUGtPzFMC_Tdx4sfmRw/s1600/img2.png" height="220" width="400" /></span></a></div>
<div class="separator" style="clear: both; text-align: center;">
<span style="font-family: Times, Times New Roman, serif;"><br /></span></div>
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<span style="font-family: Times, Times New Roman, serif;">Next steps this week will be trying to add "the * feature", adding support in <a href="http://mozilla.github.io/telemetry-dashboard/docs.html#Telemetry">telemetry.js</a> for aggregating all data over all versions and some other details that at this point make sense but don't in the context of multiple versions (histogram/summary).</span><br />
<span style="font-family: Times, Times New Roman, serif;"> If I get the time I will also like to investigate why in some cases(maybe because of the data) d3 crashes.</span><br />
<br />Ralucahttp://www.blogger.com/profile/07875649704206517173noreply@blogger.com0tag:blogger.com,1999:blog-5692089481947835006.post-18145930636178016992014-05-05T16:01:00.000-07:002014-05-05T16:01:28.681-07:00Teh foxy telemetry dashboard...a first week at Mozilla<h4>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">My story starts with the Telemetry Dashboard at Mozilla...</span></span></h4>
<br />
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">I just started working as an intern at Mozilla for the Performance team and my first task was to improve and add some features to the Telemetry Dashboard.</span></span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggM0oLjdTFalhvEpdLcPCEkB44oO7h-baxA1lE0_rh3_XAT-sTAMrgrpFDEWXm7JcQjwGniCgkqXguWKqE1ykmRAog9q9wwNl6y-DTxTSlNnhEKQ4-QbdB790M_5ChucZ6ASSN9gxrhCg/s1600/Screen+Shot+2014-05-05+at+11.34.23+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggM0oLjdTFalhvEpdLcPCEkB44oO7h-baxA1lE0_rh3_XAT-sTAMrgrpFDEWXm7JcQjwGniCgkqXguWKqE1ykmRAog9q9wwNl6y-DTxTSlNnhEKQ4-QbdB790M_5ChucZ6ASSN9gxrhCg/s1600/Screen+Shot+2014-05-05+at+11.34.23+AM.png" height="261" title="dashboard" width="400" /></a></span></span></div>
<br />
<br />
<h3>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">Keeping the story short</span></span></h3>
<br />
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">Telemetry submissions gather into large files and are then uploaded to
S3. The analysis framework then downloads them, unpacks them and runs
an aggregation over the submissions, on file at the time.
</span></span><br />
<div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">When
a set of compressed files from S3 containing telemetry submission have
been aggregated. The aggregates are then merged into the a public S3
bucket.</span></span></div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">
</span></span><div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"> </span></span></div>
<div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">The
telemetry dashboard fetches the aggregated results from the public S3
bucket. To make it easy to consume this in a browser, everything is
stored in JSON, but for storage efficiency reasons (and bandwidth) the
storage format is not very intuitive.</span></span></div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">
</span></span><div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"><br /></span></span></div>
<div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">To allow people to access the data without having to
read a complicated JSON format one can use <a href="http://telemetry.mozilla.org/docs.html#Telemetry">telemetry.js</a>. The idea is that people who write dashboards and
present the results of the aggregated data should access it through
this library. That way, if ever wanted
one can change the serverside storage format (this is something we will be
doing) and still have a working dashboard.</span></span></div>
<div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"> </span></span></div>
<div>
<h3>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">Usability, use cases and feature requests</span></span></h3>
<h3>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"> </span></span></h3>
</div>
<div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">Feature request one:</span></span></div>
<div style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">To see aggregated results one has to specify channel, version and measure.</span></span></div>
<div style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">There are two plots on the side, the top one shows evolution of the measure over "build dates" or "calendar dates". It does so by plotting mean, submissions, and percentiles over time.</span></span></div>
<div style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">However, there is currently no way to show the evolution over multiple versions.</span></span></div>
<div style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"> </span></span></div>
<div style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">Feature request two:</span></span></div>
<div style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">It would be useful to have the latest build version as a default one so there wouldn't be an additional step of selecting from all the nightly versions to get the latest updates on the current version.</span></span></div>
<div style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"> </span></span></div>
<div style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">Feature request three:</span></span></div>
<div style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">Get all the the selected options as parameters in the url.</span></span></div>
<div style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"> </span></span></div>
<div style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">Feature request four:</span></span></div>
<div style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">Add checkAll/uncheckAll buttons so you don't need to select all the features by serial check.</span></span></div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"></span></span><div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"> </span></span></div>
<div>
<h3>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">First week updates</span></span></h3>
<h3>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"> </span></span></h3>
</div>
<div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">As I started looking at the code from dashboard.js and after talking to my mentor we got to the conclusion that some major changes might be needed as the dashboard itself is designed to support only one version at a time.</span></span></div>
<div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">I started looking for some open source libraries (a nice post on this topic can be found <a href="http://home.kairo.at/blog/2014-05/finding_an_openly_licensed_js_graph_lib">here</a>) and played with javascript as it is new to me. I looked at <a href="http://dygraphs.com/">dyghaph library</a> and it seemed like a reasonable one to use.</span></span></div>
<div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">I started by changing the histogram evolution diagram.</span></span></div>
<div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"> </span></span></div>
<div class="separator" style="clear: both; text-align: center;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMZswzI4R8BjpBetIcdyEyfYgcU26ltM8kRUGIGQg_PAxeJHkcSnhQXHopG1uyYBph8N921nTmTy81TK2ipFvtUukryYhHEW-veiFHjVxXDrmJsM0LXWxe7QYpwSCChhXUbLNsKcNwX9Y/s1600/Screen+Shot+2014-05-05+at+1.57.11+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMZswzI4R8BjpBetIcdyEyfYgcU26ltM8kRUGIGQg_PAxeJHkcSnhQXHopG1uyYBph8N921nTmTy81TK2ipFvtUukryYhHEW-veiFHjVxXDrmJsM0LXWxe7QYpwSCChhXUbLNsKcNwX9Y/s1600/Screen+Shot+2014-05-05+at+1.57.11+PM.png" height="173" width="320" /></a></span></span></div>
<div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"> </span></span></div>
<div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"> I also removed the range selector as the new diagram provides zoom in by selecting range.</span></span></div>
<div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"> </span></span></div>
<div class="separator" style="clear: both; text-align: center;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"><iframe allowfullscreen='allowfullscreen' webkitallowfullscreen='webkitallowfullscreen' mozallowfullscreen='mozallowfullscreen' width='320' height='266' src='https://www.blogger.com/video.g?token=AD6v5dwAJpJEOBCqTynJSvS7wGxOJumGwppsTPIyD7nVFDAjQXINYFTx1KIAkd180O6P8rGQaUAQaldloOcqjZ4n9A' class='b-hbp-video b-uploaded' frameborder='0'></iframe></span></span></div>
<div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"><br /></span></span></div>
<div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">Next step was to dynamically add checkboxes for selecting the specific features that we need to plot ..(that made me learn about how damn neat jquery can be :p)..it is not that pretty yet but it works.</span></span></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiz3dZGrR4KlIM1SJYiNWhvG3jeGLpwFXcLIfDFjA1I-PNC2BX-nL4YjMKg99mMxdxv8itoVSV_OhyphenhyphenldhoheO8tvRjFtCmg6-N38XmDnMts1-IYBBeIClpdhYf6QfjN8iBuicDgfCLK064/s1600/checkbox.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiz3dZGrR4KlIM1SJYiNWhvG3jeGLpwFXcLIfDFjA1I-PNC2BX-nL4YjMKg99mMxdxv8itoVSV_OhyphenhyphenldhoheO8tvRjFtCmg6-N38XmDnMts1-IYBBeIClpdhYf6QfjN8iBuicDgfCLK064/s1600/checkbox.jpg" height="246" width="400" /></a></span></span></div>
<div>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"> </span></span></div>
<br />
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">I added two buttons so that checking/unchecking can be a click away for the all features selected/unselected.</span></span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmzxjJYztCroMydrXsv8aMUdQuoB0NUICcmIhRkAVAe065flx6YKoe-exzyMd3I0GrKeReGCNZVeEev_g48OtDp95xviR-rGfz-0i5C6Y2m4JRGv5NbH3vFBEKEPj6xqZGGod7iADOySk/s1600/buttons.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmzxjJYztCroMydrXsv8aMUdQuoB0NUICcmIhRkAVAe065flx6YKoe-exzyMd3I0GrKeReGCNZVeEev_g48OtDp95xviR-rGfz-0i5C6Y2m4JRGv5NbH3vFBEKEPj6xqZGGod7iADOySk/s1600/buttons.png" height="141" width="400" /></a></span></span></div>
<br />
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">By Friday I added data(test data) for several versions on the graph. (because of same duplicated series given as input the graph looks like as drawing only measures for one version but actually, if you are looking at the labels, two identical are present). </span></span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiy643qDfBY18SXKiuK3VY3QoeVIYwITqCLLN1ZmjqyLszppwn_4E_xr3ZVdlvVkEDAPn2SUyP1G02dbCKGpY16P0HwnmVy9rZ9H9YdjO-asX-8fkW-IdNp0LNrxwiF6BQ_8KS8OOyhf0/s1600/Screen+Shot+2014-05-05+at+3.06.22+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiy643qDfBY18SXKiuK3VY3QoeVIYwITqCLLN1ZmjqyLszppwn_4E_xr3ZVdlvVkEDAPn2SUyP1G02dbCKGpY16P0HwnmVy9rZ9H9YdjO-asX-8fkW-IdNp0LNrxwiF6BQ_8KS8OOyhf0/s1600/Screen+Shot+2014-05-05+at+3.06.22+PM.png" height="318" width="400" /></a></span></span></div>
<br />
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">Bottom line: I started doing some neat stuff last week an Mozilla aaaand got a foto with the Fox at the <span class="tl"><em><a href="https://webwewant.mozilla.org/en/#video">Firefox 29</a>release :D</em></span></span></span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZWPeDk6l8Phi570Z4na1m2CstnJDitCP2HzeOowt8r3A8rLowak9NB5PLNR2IVQI9iWxmyOt2Lf7kYoVeY48XTD9erzA89TJr4JbGKpm_XAaIEYHkHUQY-yIkoLxmbNjuiEUPdLoeYfk/s1600/IMG_20140429_164743.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZWPeDk6l8Phi570Z4na1m2CstnJDitCP2HzeOowt8r3A8rLowak9NB5PLNR2IVQI9iWxmyOt2Lf7kYoVeY48XTD9erzA89TJr4JbGKpm_XAaIEYHkHUQY-yIkoLxmbNjuiEUPdLoeYfk/s1600/IMG_20140429_164743.jpg" height="300" width="400" /></a></span></span></div>
<br />
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"><span class="tl"><em><br /></em></span></span></span>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"> This week I will be refactoring the code and I will implement the missing parts. I am pretty sure I've got a loooooot to work/learn this week :D.</span></span><br />
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;">BTW my live playground is hosted <a href="http://raluca-elena.github.io/telemetry-dashboard/#nightly/25/COOKIES_3RDPARTY_NUM_ATTEMPTS_ACCEPTED">here</a>.</span></span><br />
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"> </span></span><br />
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"><br /></span></span>
<span style="font-size: small;"><span style="font-family: Georgia,"Times New Roman",serif;"> </span></span><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Ralucahttp://www.blogger.com/profile/07875649704206517173noreply@blogger.com0San Francisco, CA, USA37.7749295 -122.4194155000000137.373502 -123.06486250000002 38.176356999999996 -121.77396850000001