index.html


<!DOCTYPE html>
<!--[if IEMobile 7 ]><html class="no-js iem7"><![endif]-->
<!--[if lt IE 9]><html class="no-js lte-ie8"><![endif]-->
<!--[if (gt IE 8)|(gt IEMobile 7)|!(IEMobile)|!(IE)]><!--><html class="no-js" lang="en"><!--<![endif]-->
<head>
  <meta charset="utf-8">
  <title>Manufacturing Big Data</title>
  <meta name="author" content="System Insights">

  
  <meta name="description" content="Industrial energy consumers in Tamil Nadu, India have seen a sharp increase in energy costs beginning from April 2012. In this post we look at the &hellip;">
  

  <!-- http://t.co/dKP3o1e -->
  <meta name="HandheldFriendly" content="True">
  <meta name="MobileOptimized" content="320">
  <meta name="viewport" content="width=device-width, initial-scale=1">

  
  <link rel="canonical" href="http://www.manufacturingbigdata.com/">
  <link href="/favicon.png" rel="icon">
  <link href="/stylesheets/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
  <script src="/javascripts/modernizr-2.0.js"></script>
  <script src="/javascripts/ender.js"></script>
  <script src="/javascripts/octopress.js" type="text/javascript"></script>
  <link href="/atom.xml" rel="alternate" title="Manufacturing Big Data" type="application/atom+xml">
  <!--Fonts from Google"s Web font directory at http://google.com/webfonts -->
<link href="http://fonts.googleapis.com/css?family=PT+Serif:regular,italic,bold,bolditalic" rel="stylesheet" type="text/css">
<link href="http://fonts.googleapis.com/css?family=PT+Sans:regular,italic,bold,bolditalic" rel="stylesheet" type="text/css">

  
</head>

<body   >
  <header role="banner"><hgroup>
  <h1><a href="/">Manufacturing Big Data</a></h1>
  
    <h2>System Insights Corporate Blog</h2>
  
</hgroup>

</header>
  <nav role="navigation"><ul class="subscription" data-subscription="rss">
  <li><a href="/atom.xml" rel="subscribe-rss" title="subscribe via RSS">RSS</a></li>
  
</ul>
  
<form action="http://google.com/search" method="get">
  <fieldset role="search">
    <input type="hidden" name="q" value="site:www.manufacturingbigdata.com" />
    <input class="search" type="text" name="q" results="0" placeholder="Search"/>
  </fieldset>
</form>
  
<ul class="main-navigation">
  <li><a href="/">Blog</a></li>
  <li><a href="/blog/archives">Archives</a></li>
</ul>

</nav>
  <div id="main">
    <div id="content">
      <div class="blog-index">
  
  
    <article>
      
  <header>
    
      <h1 class="entry-title"><a href="/blog/2012/07/10/tneb-regulations/">Energy Regulations in Tamil Nadu, India</a></h1>
    
    
      <p class="meta">
        

<time datetime="2012-07-10T11:00:00+05:30" pubdate data-updated="true">Jul 10<span>th</span>, 2012</time>
        
      </p>
    
  </header>


  <div class="entry-content"><p>Industrial energy consumers in Tamil Nadu, India have seen a sharp increase in energy costs beginning from April 2012. In this post we look at the revised tariff of the Tamil Nadu Electricity Board (TNEB), and examine its impact on the overall energy costs for a manufacturing plant.</p>

<h2>The Tariff</h2>

<p>The revised Tariff for Industrial Consumers (HT1A) is as follows (available <a href="http://tnerc.tn.nic.in/orders/Tariff%20Order%202009/2012/T.O%20No.%201%20of%202012%20dated%2030-03-2012.pdf">here</a>):</p>

<h3>Basic Charges</h3>

<ul>
<li>Demand Charges &#8211; INR 300 / kVA / month</li>
<li>Energy Charges &#8211; INR 5.50 / kWh</li>
</ul>


<h3>Restrictions and Surchages</h3>

<h4>Power Factor:</h4>

<ul>
<li>Power Factor => 0.9 &#8211; No Surcharge</li>
<li>0.9 > Power Factor => 0.85    &#8211; 1% of Current Consumption Charges for every 0.01 reduction in PF from 0.9</li>
<li>0.85 > Power Factor => 0.75 &#8211; 1.5% of Current Consumption Charges for every 0.01 reduction in PF from 0.9</li>
<li>Power Factor &lt; 0.75 &#8211; 2% of Current Consumption Charges for every 0.01 reduction in PF from 0.9</li>
</ul>


<h4>Billable Demand:</h4>

<ul>
<li>Demand Charges will levied on Maximum Demand that has actually been registered for the month or 90% of the Sanctioned Demand, whichever is higher.</li>
</ul>


<h4>Peak Hour:</h4>

<ul>
<li>HT Industrial Consumers will be billed 20% extra on the Energy Charges for the Energy recorded during the Peak hours</li>
<li>Duration of Peak hours will be 6:00am to 9:00am &amp; 6:00pm to 9:00pm</li>
</ul>


<h4>Night Hour:</h4>

<ul>
<li>HT Industrial Consumers will get a reduction of 5% on the Energy Charges for the Energy recorded during the Night hours</li>
<li>Duration of Night hour will be 10:00pm to 5:00am</li>
</ul>


<h4>Demand Integration Period:</h4>

<ul>
<li>Maximum Demand Integration period will be 15minutes</li>
</ul>


<h4>Harmonics:</h4>

<ul>
<li>Total Voltage Harmonic Distortion should not exceed 5%</li>
<li>Total Current Harmonic Distortion should not exceed 8%</li>
<li>If the harmonics level are not within the limits, then the consumer has to pay 15% of respective tariff as Compensation</li>
</ul>


<h2>Power Scenario in the State</h2>

<p>The state of Tamil Nadu has installed capacity of 10,364.5 MW and the average power availability is about 8500 MW. The demand ranges from 11,500 MW to 12,500 MW which gives a clear indication that the state has a shortage about 3000 to 4000 MW of power. Moreover if we see the growth of consumers, it keeps increasing at a rate of 5% every year. Because of this gap between demand and supply, TNEB has taken the following mitigation measures:</p>

<ul>
<li>40% cut on demand and energy for High Tension Industrial and Commercial Services</li>
<li>Load shedding of 2 hrs in Chennai and its suburbs</li>
<li>Load shedding of 4 hrs in other urban and rural areas</li>
<li>10% of power supply during Peak hours for Industrial and Commercial Services</li>
<li>Power Holiday for all HT &amp; LT consumers</li>
</ul>


<p>These restrictions can be relaxed based on the power availability. However, HT consumers are allowed to make power purchase to inter- and intra-state Open Access providers where cheaper power may be available.</p>

<h2>So&#8230; What&#8217;s the Impact?</h2>

<p>Lets examine how these revised prices impact an average manufacturing plant.</p>

<p>Lets consider a Manufacturing Facility, with the following cost structure:</p>

<ul>
<li>Permitted Demand: 1000 kVA</li>
<li>Permitted Energy Quota: 300,000 kWh</li>
</ul>


<p>Based on this structure, lets assume that the energy consumed and the costs incurred during a representative month is as follows:</p>

<iframe width='410' height='500' frameborder='0' src='https://docs.google.com/spreadsheet/pub?key=0AjFwRioMlxbbdEptNTRoLVdtQUpsa1pIMW9NSXl5S1E&single=true&gid=0&output=html&widget=true'></iframe>


<p>Lets see how costs change with the new pricing under different scenarios:</p>

<h3>Case 1: Grid Only</h3>

<p>The revised grid costs are Rs. 5.50/kWh and the plant faces a 40% reduction on its demand and energy limits. The revised permitted demand is 600 kVA and the permitted energy is 180,000 kWH. The plant is penalized at twice the price for exceeding the energy or demand limits.</p>

<iframe width='410' height='600' frameborder='0' src='https://docs.google.com/spreadsheet/pub?key=0AjFwRioMlxbbdEptNTRoLVdtQUpsa1pIMW9NSXl5S1E&single=true&gid=1&output=html&widget=true'></iframe>


<p>If the plant is purely dependent on the grid (EB), then monthly energy costs grow by more than 120% (more than doubles!!!).</p>

<h3>Case 2: Grid and Diesel</h3>

<p>If the plant offsets 500 kVA of demand and 130,000 kWh of energy by running a diesel generator, which costs Rs. 15/kWh:</p>

<iframe width='410' height='650' frameborder='0' src='https://docs.google.com/spreadsheet/pub?key=0AjFwRioMlxbbdEptNTRoLVdtQUpsa1pIMW9NSXl5S1E&single=true&gid=2&output=html&widget=true'></iframe>


<p>Even with using an auxilliary Diesel Generator to supplement grid energy, the plant spends 82% more on energy.</p>

<h3>Case 3: Grid and Power Purchase</h3>

<p>The plant purchases 130,000 kWh at Rs. 8/kWh, and gets a equivalent deemed demand of 260 kVA:</p>

<iframe width='410' height='700' frameborder='0' src='https://docs.google.com/spreadsheet/pub?key=0AjFwRioMlxbbdEptNTRoLVdtQUpsa1pIMW9NSXl5S1E&single=true&gid=3&output=html&widget=true'></iframe>


<p>The plant still sees an increase of about 60% after purchasing power from third party suppliers.</p>

<h2>What do we do?</h2>

<p>Its clear that there are no simple ways of reducing or even maintaining energy costs at the &#8220;pre-hike&#8221; levels in Chennai. Simply changing the energy source to Diesel is not an option either, and buying third-party power can be just as expensive as using energy from the TNEB grid. What this calls for is more aggressive and hands-on management of energy consumption in the manufacturing facility, looking at which machines and systems consume the most energy, and finding ways to decrease their usage. Our vimana platform does just this and we will be back with a followup post on how <a href="http://www.systeminsights.com/vimana">vimana</a> can be applied in improving energy efficiency and reducing energy costs in a manufacturing facility.</p>
</div>
  
  
    </article>
  
  
    <article>
      
  <header>
    
      <h1 class="entry-title"><a href="/blog/2012/07/02/monitoring-mtc-streams-mtconnect-graphr/">Monitoring MTConnect Streams: MTConnect Graphr</a></h1>
    
    
      <p class="meta">
        

<time datetime="2012-07-02T11:15:00+05:30" pubdate data-updated="true">Jul 2<span>nd</span>, 2012</time>
        
      </p>
    
  </header>


  <div class="entry-content"><p>A few months back I took up the fun task of exploring MTConnect streams and the amazing possibilities which it presented to a developer. That culminated into a web base monitoring app, MTConnect Graphr, which can now be downloaded from <a href="http://github.com/princearora/mtconnect-graphr">Github</a>. In this post I&#8217;ll run down through the development process of the same.</p>

<p>The web scene has changed remarkably in a past few years. Web applications are expected to be compatible to smartphones and tablets. They are supposed to be clean and responsive. This requirement was enough to persuade me to use <a href="http://twitter.github.com/bootstrap/">Bootstrap by Twitter</a> as the basic framework for the app. It is tiny, expandable and has good documentation to get you started.</p>

<p>Coming to the implementation, the first task was to dynamically connect to the XML stream provided by the MTConnect Agent. Though it sounds pretty easy, the task becomes a bit tricky because it requires a recurring connection with external urls. The easiest way out of this situation is to implement a PHP proxy. For this we write a PHP loader script, and then use it every time we need to connect to an external host.</p>

<pre><code>header('Content-type: application/xml'); //specififying the return content type
$q = $_GET['url'];
  handle = fopen($q, "r");              //connecting to the url
if ($handle) {                          
    while (!feof($handle)) {
        $buffer = fgets($handle, 4096);
        echo $buffer;                  //reading and returning content
    }
    fclose($handle);
}
</code></pre>

<p>That sums up our <code>loader.php</code>. Next all we need to do is to write a simple function to load the XML file via the proxy.</p>

<pre><code>function getCurrentXML(conn_url) {              //function to retrieve a xml file 
    var n = "";
    return $.ajax({
        url: "loader.php?url=http://"+conn_url, //using the proxy
        cache: !1,
        async: !1,
        dataType: "txt",
        success: function (t) {
            n = t
        }
    }), n
}
</code></pre>

<p>Once we have access to the XML stream, there are a plethora of tools available to parse and get data out of it. I chose to use a combination of jQuery and JSON for the job. The xml2json plugin available <a href="http://www.fyneworks.com/jquery/xml-to-json/">here</a> provided an easy conversion to JSON.</p>

<pre><code>var xmldata = getCurrentXML(conn_url),
    i = $.xml2json(xmldata);
</code></pre>

<p>With JSON, life is easy. It can&#8217;t be any simpler to parse data than it is with JSON. All I did was to write regular funtions to parse conditions and device parameters. But there is a catch. Not all XML tags will be meaningful and to make them appear right, we need to write individual functions for each of them. Here I will explain the working of the function used to parse the conditions for all parameters.</p>

<pre><code>this.getCondition = function (n) {
  var r=new Object();
  r.type= new Array(),r.value=new Array();
  var count=0;
  for (var t = 0; t &lt; n.ComponentStream.length; t++) {
    var i = n.ComponentStream[t],
        u = i.name;
    if (i.Condition)
    {
      var v = i.Condition;
      if(v.Normal){ if(v.Normal.length&gt;1){
        for(var f =0; f &lt; v.Normal.length; f++)
        {
          r.type[count] =  u+' '+v.Normal[f].type;
          r.value[count++] = "Normal"
        }}
      else{
          r.type[count] =  u+' '+v.Normal.type;
          r.value[count++] = "Normal"}
        }
      ..........
      //similarly for other conditions 
     }r.len=count;
   }
       return r 
},
</code></pre>

<p>Though this takes away the reusability of the script, the data displayed turns out to be easier to comprehend.</p>

<p>With all these pieces in place, a simple requirement is to refresh the input from the MTC stream every few milliseconds. A recursive function with delay takes care of that</p>

<pre><code>var updateFromMTC = function(){
.....
  setTimeout('updateFromMTC("'+conn_url+'")' , 1000);
.....
}
</code></pre>

<p>The task we confront next is to display it elegantly. That is taken care by using power of HTML5. The devices in the stream are populated at the top of the page with the color of each dependent on the availability of the device. To display all the parameters we use two empty divs, in which the parameters and the conditions are populated using a javascript user function.</p>

<pre><code>&lt;div class="container" id='SelStats' position='absolute'"&gt;
  &lt;div id='conditions' position='absolute'&gt;&lt;/div&gt;
&lt;/div&gt;

var updateSelected = function(){
 if(thePage.ActiveShape){
   var selShp = thePage.ActiveShape;
    var SelDisplay = document.getElementById('SelStats');
    if(SelDisplay &amp;&amp; selShp){
     if(selShp.deviceName != ''){
        $(SelDisplay).empty();
        $(SelDisplay).append('&lt;b&gt;Machine:&lt;/b&gt;' + selShp.text);
        ......
</code></pre>

<p>So, this completes the basic task of monitoring an MTConnect stream. Next we need to plot it. There are some really advanced open source scripts out there to assist plotting data, but for this particular task <a href="http://smoothiecharts.org/">Smoothie Charts</a> seemed a perfect fit to me. It is a really small charting library designed for live streaming data. Integration with the existing code was easy. A few more lines to the code, and it plots like a charm.</p>

<pre><code>var smoothie = new SmoothieChart();
  smoothie.streamTo(document.getElementById("mycanvas") 3000 /*delay*/);
var line1 = new TimeSeries();
  setInterval(function() {line1.append(new Date().getTime(), math.random());}, 3000 /*delay*/);
</code></pre>

<p>Finally, we need to add an emergency alarm light for the parameter being monitored. A slick form to enter the maximum/minimum value, and a basic function to compare instantaneous values are enough to pull it off. With the div being populated dynamically every few seconds, we need to save some info in a cookie which is made easy by the <a href="https://github.com/carhartl/jquery-cookie/">jQuery-cookie</a> plugin.</p>

<div style="text-align: center;">
<img src="/images/graphr-1.jpg" width=360 height=600 /> <img src="/images/graphr-2.jpg" width=360 height=600 /> </div>


<p>I guess that&#8217;s it. The app is ready to roll. I checked it out locally on a PC, iPOD touch and an android device. Seems to be working fine for me. Let me know if any of you notice anything off about it.</p>

<p>PS: Please ensure that the application is run on a PHP server. Otherwise the application will fail to connect to the stream and all you will see is a white blank page. I&#8217;d recommend WAMP/LAMP for users trying it on their personal PCs.</p>
</div>
  
  
    </article>
  
  
    <article>
      
  <header>
    
      <h1 class="entry-title"><a href="/blog/2012/06/13/s3/">Octopress + S3 + Cloudfront</a></h1>
    
    
      <p class="meta">
        

<time datetime="2012-06-13T07:03:00+05:30" pubdate data-updated="true">Jun 13<span>th</span>, 2012</time>
        
      </p>
    
  </header>


  <div class="entry-content"><p>We have finally migrated to Octopress from Google Blogger. Feels a lot better to write in Markdown than using Blogger&#8217;s editor! The static content is being stored in an S3 bucket and served through AWS Cloudfront.</p>

<p>Thank you: <a href="http://www.jerome-bernard.com/blog/2011/08/20/quick-tip-for-easily-deploying-octopress-blog-on-amazon-cloudfront/">Jerome Bernard</a>, <a href="http://blog.jacobelder.com/2012/03/octopress-and-cloudfront/">Jacob Elder</a>, <a href="http://www.octopress.org">Octopress</a>, and <a href="http://www.github.com">Github</a>.</p>
</div>
  
  
    </article>
  
  
    <article>
      
  <header>
    
      <h1 class="entry-title"><a href="/blog/2012/06/08/why-saas/">Why SaaS?</a></h1>
    
    
      <p class="meta">
        

<time datetime="2012-06-08T11:16:00+05:30" pubdate data-updated="true">Jun 8<span>th</span>, 2012</time>
        
      </p>
    
  </header>


  <div class="entry-content"><p>A common question that we always get is why vimana is a software-as-a-service (SaaS) product. Surely, given how much data we are collecting, it must be easier to run it locally inside a plant, right? Well, if all vimana was doing was creating plots of part counts and utilization, then yes, running it locally does make sense. But vimana does a lot more &#8211; it helps understand the patterns behind productivity (and the lack thereof), and being able to support these capabilities requires a whole lot more of computational resources.</p>

<p>So lets dig deeper &#8211; why SaaS?</p>

<ol>
<li><strong>Keep it Growing</strong>: SaaS allows us to scale product functionality as your operations grow. This means that vimana can scale to support an increasing number of devices, along with the analytical capabilities required to support them. SaaS also allows us to keep the app at the latest version without requiring long downtimes for the updates.</li>
<li><strong>Keep it All</strong>: SaaS allows us to keep historical plant data securely for as long as you want us to. This makes it possible to baseline against historical data to put current performance in context, and to make better decisions about the future based on past usage and operational patterns.</li>
<li><strong>Keep it Lean</strong>: SaaS enables simple, annual, pay-as-you-go pricing, where you pay based on the number of devices you have connected to vimana, and the kind of analysis being performed on the devices. Since the app is delivered over the web, any number of users can access it (even simultaneously!).</li>
</ol>


<p>SaaS deployments also allow us to farm out specific analytical processes to elastic clusters, using map reduce and other big-data-crunching technologies. We will be talking about this in detail in upcoming posts.</p>
</div>
  
  
    </article>
  
  
    <article>
      
  <header>
    
      <h1 class="entry-title"><a href="/blog/2012/06/08/livebetter-article/">BEC Article in Livebetter Magazine</a></h1>
    
    
      <p class="meta">
        

<time datetime="2012-06-08T08:23:00+05:30" pubdate data-updated="true">Jun 8<span>th</span>, 2012</time>
        
      </p>
    
  </header>


  <div class="entry-content"><p>The June issue of the <a href="http://livebettermagazine.com">Livebetter Magazine</a> features an article about the BEC standard by Ralph Resnick from the National Center for Defense Manufacturing and Machining and myself. You can read the article <a href="http://livebettermagazine.com/eng/magazine/article_detail.lasso?id=307">here</a>.</p>
</div>
  
  
    </article>
  
  
    <article>
      
  <header>
    
      <h1 class="entry-title"><a href="/blog/2012/05/30/powered-by-octopress/">Powered by Octopress</a></h1>
    
    
      <p class="meta">
        

<time datetime="2012-05-30T10:00:00+05:30" pubdate data-updated="true">May 30<span>th</span>, 2012</time>
        
      </p>
    
  </header>


  <div class="entry-content"><p>We are moving to the Octopress framework. The blog will be hosted on Github Pages.</p>
</div>
  
  
    </article>
  
  
    <article>
      
  <header>
    
      <h1 class="entry-title"><a href="/blog/2012/05/08/mongo-replica-sets/">MongoDB and Replica Sets</a></h1>
    
    
      <p class="meta">
        

<time datetime="2012-05-08T10:50:00+05:30" pubdate data-updated="true">May 8<span>th</span>, 2012</time>
        
      </p>
    
  </header>


  <div class="entry-content"><h2>Why do we care</h2>

<p>We use mongoDB to persist data from the vimana tenants. All historical data from our customers is stored in mongo, and its very important that we can continuously persist into mongo without losing data. We are using replica sets to make sure that our data is redundantly stored and to ensure that we have a failover mechanism when one of the mongoDBs lose connection.</p>

<h2>Introduction to Replica Sets</h2>

<p>Replica sets are a form of asynchronous master/slave replication, adding automatic failover and automatic recovery of member nodes. A replica set consists of two or more nodes that are copies of each other. (i.e.: replicas). The replica set automatically elects a primary (master). Drivers (and mongos) can automatically detect when a replica set primary changes and will begin sending writes to the new primary.</p>

<h2>Those GOTCHAs</h2>

<p>In order enable replica sets, you need to pass the &#8220;replSet&#8221; parameter while starting the mongod processes.
Replica sets cannot be initiated on those mongod instance which were already started with out &#8220;replSet&#8221; parameter.  Stop and restart the mongod processes with replSet parameter.
When replica sets are configured, all the writes will only go to the primary!!! Mongo has its own algorithms for syncing the data across the nodes.</p>

<h2>That setup</h2>

<p>Starting multiple mongo instances.</p>

<pre><code>$ mongod --dbpath mongo_rpl/data1 --replSet set1 --port 27018
$ mongod --dbpath mongo_rpl/data2 --replSet set1 --port 27019
$ mongod --dbpath mongo_rpl/data3 --replSet set1 --port 27020
</code></pre>

<p>This starts 3 different instances of mongo running on different ports.</p>

<h2>Setting up the replica config</h2>

<pre><code>➜  ~  mongo localhost:27018
MongoDB shell version: 1.8.2
Mon Apr 30 11:13:59 *** warning: spider monkey build without utf8 support.  consider rebuilding with utf8 support
connecting to: localhost:27018/test
&gt; config = {_id: "set1", members: [{_id: 0, host: "localhost:27018"}, {_id: 1, host: "localhost:27019"},{_id: 2, host:"localhost:27020"}]}
{
   "_id" : "set1",
  "members" : [
 {
    "_id" : 0, 
   "host" : "localhost:27018"
},
{
   "_id" : 1,
  "host" : "localhost:27019"
},
{
   "_id" : 2,
  "host" : "localhost:27020"
}
]
}
&gt; rs.initiate(config)
{
   "info" : "Config now saved locally.  Should come online in about a minute.",
  "ok" : 1
}
&gt;
</code></pre>

<p>This config enables the replicas to talk with each other. The talking would involve the replicas communicating who is the primary and who are all the secondaries.</p>

<h2>That app</h2>

<p>We wrote a <a href="https://github.com/deepakprasanna/mongo_oplog_watcher">small app</a> which would tail the oplog of our test servers and insert some interesting records into the replicas.</p>

<h2>Those observations</h2>

<p>The most interesting part of playing with replica sets is to understand how mongoDB behaves and handles a node failure.
I have classified my observations into two catagories, a node failure during a read and during a write.</p>

<h3>Reading Scenarios</h3>

<p>MongoDB doesnot by default support serving reads from the replicas. MongoDB will serve all reads from primary by default. Reads from secondaries can be configured, which is actually a 2 step process.  The first step is to configure &#8220;slaveOK&#8221; in the mongo console. This will tell mongo, it is okay to serve the reads from the secondaries. The second step is to instantiate the MongoReplConnection with :read => :secondary option.
This will tell the driver that it is okay to send the reads to the secondaries. Mongo driver will randomly select one of the secondaries to serve the reads. The distribution of the reads across the secondaries is handled by the driver.</p>

<ul>
<li><p><strong>Secondary goes down:</strong> <br/>
<strong>slaveOk:</strong> <code>Mongo::ConnectionFailure</code> will be raised when there is a failure. Mongo driver is intelligent, when it sees a <code>Mongo::ConnectionFailure</code> it prevents the next reads from going to that dead secondary.
The driver has its own algorithm to find out if the dead secondary is back alive or not. As far a read is concerned we need to catch <code>Mongo::ConnectionFailure</code> and make the read once more(Assuming that another secondary will be up).
If the secondaries are configured to serve the reads, then the primary is not touched at all until all other secondaries are dead. But there is no real way to find out which mongod instance served the read. <br/>
<strong>Without SlaveOk:</strong>  Rest of the world goes as usual.</p></li>
<li><p><strong>Primary goes down:</strong>  <br/>
<strong>slaveOk:</strong> Rest of the world goes as usual. <br/>
<strong>without SlaveOk:</strong> All the reads are going to fail since primary can only serve the reads. There are 2 ways to solve this problem, catch the exception and throw an error message. Or keep polling the server until one of the other secondaries becomes a primary and read becomes successful. If the client decides to retry, it&#8217;s not guaranteed that another member of the replica set will have been promoted to primary right away, so it&#8217;s still possible that the driver will raise another <code>Mongo::ConnectionFailure</code>.</p></li>
</ul>


<h3>Writing scenarios</h3>

<ul>
<li><strong>Secondary goes down:</strong> When replica sets are configured in mongoDB, all the writes go to the primary. So there is no problem at all if a secondary goes down.
If the secondary comes up again, mongo will take care of replicating the records which were lost by the time when it was down. Perfect!</li>
<li><strong>Primary goes down:</strong> The ruby mongo driver raises <code>Mongo::ConnectionFailure</code>. Checkout the oplog watcher, we have caught this exception and we are doing a <code>puts</code> that the connection is lost.
  But how ever the after a few seconds, when one of the other secondaries get elected as the primary the writes become successful. The interesting fact is that all the writes which failed
  during this recovery process is lost. Since we are able to catch <code>Mongo::ConnectionFailure</code>, it is up-to the client to pull the sleeves and persist the data somewhere else until another
  secondary becomes a primary.While testing we lost about 20-30 records when the primary was down. I guess this number would be more or less depending the latency which we would face in the realtime.</li>
<li><strong>Last mongod instance is not becoming primary:</strong> As you can see from the config above, we have 3 replicas. So we would have one primary(27018) and two secondaries(27019 and 27020). We are stopping the primary(27018). Now one of the secondaries becomes a primary(say 28019). Now we would have one primary(27019) and one secondary(27020). Again we are stopping the primary(27019). But the left over secondary(27020) will not become the primary!!!!!!! This causes all the writes to fail. But if we bring another dead mongod instance(27018) up, the leftover secondary(27020) becomes the primary and from then on the writes become successful.  This was found while digging into the mongo logs.</li>
</ul>


<p><code>[rs Manager] replSet can't see a majority, will not try to elect self:</code></p>

<p>From what I understand , Replica sets will do the election process only if there are 2 or more replicas available. If at all we have only one replica alive, the election will not happen and all the writes will fail because there will be no primary. We need have 2 replicas alive at anytime for the writes to become successful. This is interesting.</p>

<p>Happy hacking, <br/>
Deepak.</p>
</div>
  
  
    </article>
  
  
    <article>
      
  <header>
    
      <h1 class="entry-title"><a href="/blog/2012/04/14/welcome-deepak/">Welcome Deepak!</a></h1>
    
    
      <p class="meta">
        

<time datetime="2012-04-14T10:48:00+05:30" pubdate data-updated="true">Apr 14<span>th</span>, 2012</time>
        
      </p>
    
  </header>


  <div class="entry-content"><p>We would like to welcome Deepak Prasanna to System Insights. Deepak starts this month as a Software Developer working from our Chennai office. You will be hearing from him soon in this blog.</p>
</div>
  
  
    </article>
  
  
    <article>
      
  <header>
    
      <h1 class="entry-title"><a href="/blog/2012/03/21/McKinsey-Part-2-What-do-do-with-it/">McKinsey on (Manufacturing) Big Data - Part 2 - What Do Do With It</a></h1>
    
    
      <p class="meta">
        

<time datetime="2012-03-21T13:47:00+05:30" pubdate data-updated="true">Mar 21<span>st</span>, 2012</time>
        
      </p>
    
  </header>


  <div class="entry-content"><p>We are back studying McKinsey&#8217;s report on Big Data following the last post, and here lets take a closer look at what we can do with Big Data. The report identifies several &#8220;levers&#8221; where data can be used to improve manufacturing performance (see below), and of these levers, we are primarily interested in these two:</p>

<p><img class="center" src="/images/2012-03-21-McKinsey-Part-2-What-do-do-with-it-levers.png"></p>

<ol>
<li>Implementing Lean Manufacturing (#5)</li>
<li>Using sensor data-driven operations analytics (#6)</li>
</ol>


<p>Lets look at the first of these in this post.</p>

<h2>Lean Manufacturing and the Digital Factory</h2>

<p>McKinsey identifies using Big Data to &#8220;create process transparency, develop dashboards, and visualize bottlenecks&#8221;. Our vimana application is one example of applying Big Data to create realtime dashboards of manufacturing equipment (to learn more about vimana, please visit www.systeminsights.com/vimana.) The idea here is to provide a shopfloor user both a high-level &#8220;macro&#8221; view of the shopfloor, as well as a low-level &#8220;micro&#8221; view of a single device.  Questions the dashboards can help answer include:</p>

<ul>
<li>Which devices are producing parts today?</li>
<li>Which parts are being made right now?</li>
<li>How many parts have I made?</li>
<li>What has my device been doing today?</li>
<li>What is my efficiency?</li>
<li>Why have I been in a downtime?
  (This is a much more interesting question to answer than just &#8220;What are my downtimes&#8221;. Knowing <em>why</em> downtimes occur can directly help in reducing or eliminating the downtimes, and is a step beyond simply knowing <em>that</em> a has downtime occurred).</li>
</ul>


<p>The data for vimana is streamed using the MTConnect Agent associated with each machine tool. The application itself runs in the cloud, aggregating hundreds of events every second from each machine tool. The analysis is done in realtime, and the visualizations are rendered instantly. Process transparency is achieved here because the application serves as a central repository of what is going on in the shopfloor, and multiple stakeholders (production, engineering, maintenance, management) can all use it to support decisions that come under their purview. This also takes us one step closer to the Digital Factory, where detailed operational data from the factory equipment is applied in building a complete digital model of the factory&#8217;s operations, which can then be applied in optimizing its performance.</p>

<p><em>Macro Dashboard: Shopfloor</em>
<img class="center" src="/images/2012-03-21-McKinsey-Part-2-What-do-do-with-it-dashboard.png"></p>

<p><em>Micro Dashboard: Device</em>
<img class="center" src="/images/2012-03-21-McKinsey-Part-2-What-do-do-with-it-details.png"></p>

<p>Coming back to McKinsey, they estimate a reduction of 10 to 50% in costs from applying Big Data to implement Lean Manufacturing and the Digital Factory, accompanied by a marginal increase in revenue (2%). We have already seen vimana helping improve device utilization by over 25%, which directly leads to cost reductions. The real impact of Big Data here is not as much as in enabling us to ask new questions about a shop&#8217;s productivity, but in helping us find even better answers for the same questions we have been asking for a long time and thus driving down costs. Of course, thats not to say that we cannot ask new questions based on the data –this is where data specifically from ubiquitous and low cost sensors can play a role, which we will examine in a future post.</p>
</div>
  
  
    </article>
  
  
    <article>
      
  <header>
    
      <h1 class="entry-title"><a href="/blog/2012/03/16/Setting-up-a-MTConnect-Agent-on-a-Linux-machine/">Setting Up a MTConnect Agent on a Linux (Ubuntu) Machine</a></h1>
    
    
      <p class="meta">
        

<time datetime="2012-03-16T16:10:00+05:30" pubdate data-updated="true">Mar 16<span>th</span>, 2012</time>
        
      </p>
    
  </header>


  <div class="entry-content"><p>This was blogged first at my personal blog <a href="http://princearora.wordpress.com/2012/02/24/setting-up-a-mtconnect-agent-on-a-unixubuntu-machine/">here</a>.</p>

<p>While working on my internship project, I got a chance to test out an MTConnect agent built in C++ for Linux (ubuntu). I was surprised to find that absolutely no documentation existed for setting up the Agent in a Linux environment. Although it didn&#8217;t turn out to be a big hassle in the end, I thought that it would be a good idea to list down the process of setting up a MTConnect agent on Linux. So, here you go:</p>

<ul>
<li>Download the zip archive of the latest version of MTConnect C++ Agent SDK from MTConnect Github and extract its contents onto your local disk.</li>
<li><p>Download &amp; Install libxml-2.0 and libxml-dev packages from the apt repository.</p>

<pre><code>  $ apt-get install libxml2
  $ apt-get install libxml2-dev
</code></pre></li>
<li><p>Now you need to prepare a Makefile in order to compile the agent. This can be done using the CMake package. Download &amp; install CMake if you don&#8217;t already have it.</p>

<pre><code>  $ apt-get install cmake     
</code></pre></li>
<li><p>Open the &#8216;agent&#8217; folder in the terminal and run cmake and make.</p>

<pre><code>  $ cd agent/
  $ cmake .
  $ make 
</code></pre></li>
<li><p>If everything went right, your agent would have been build. You can now start it off as a service.</p>

<pre><code>  $ ./agent daemonize 
</code></pre></li>
<li><p>If you are unsure whether the process is running, you can check out the process status:</p>

<pre><code>  $ ps aux | grep agent 
</code></pre></li>
</ul>


<p>The agent service should be up and running. You may change the agent.cfg file in any text editor based on the instructions here.</p>

<p>Have fun!</p>
</div>
  
  
    </article>
  
  <div class="pagination">
    
      <a class="prev" href="/blog/page/2/">&larr; Older</a>
    
    <a href="/blog/archives">Blog Archives</a>
    
  </div>
</div>
<aside class="sidebar">
  
    <section>
  <h1>Recent Posts</h1>
  <ul id="recent_posts">
    
      <li class="post">
        <a href="/blog/2012/07/10/tneb-regulations/">Energy Regulations in Tamil Nadu, India</a>
      </li>
    
      <li class="post">
        <a href="/blog/2012/07/02/monitoring-mtc-streams-mtconnect-graphr/">Monitoring MTConnect Streams: MTConnect Graphr</a>
      </li>
    
      <li class="post">
        <a href="/blog/2012/06/13/s3/">Octopress + S3 + Cloudfront</a>
      </li>
    
      <li class="post">
        <a href="/blog/2012/06/08/why-saas/">Why SaaS?</a>
      </li>
    
      <li class="post">
        <a href="/blog/2012/06/08/livebetter-article/">BEC Article in Livebetter Magazine</a>
      </li>
    
  </ul>
</section>


<section>
  <h1>Latest Tweets</h1>
  <ul id="tweets">
    <li class="loading">Status updating&#8230;</li>
  </ul>
  <script type="text/javascript">
    $.domReady(function(){
      getTwitterFeed("systeminsights", 4, false);
    });
  </script>
  <script src="/javascripts/twitter.js" type="text/javascript"> </script>
  
    <a href="http://twitter.com/systeminsights" class="twitter-follow-button" data-show-count="false">Follow @systeminsights</a>
  
</section>


</aside>

    </div>
  </div>
  <footer role="contentinfo"><p>
  Copyright &copy; 2012 - System Insights -
  <span class="credit">Powered by <a href="http://octopress.org">Octopress</a></span>
</p>

</footer>
  

<script type="text/javascript">
      var disqus_shortname = 'manufacturingbigdata';
      
        
        var disqus_script = 'count.js';
      
    (function () {
      var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
      dsq.src = 'http://' + disqus_shortname + '.disqus.com/' + disqus_script;
      (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
    }());
</script>


  <script type="text/javascript">
    (function(){
      var twitterWidgets = document.createElement('script');
      twitterWidgets.type = 'text/javascript';
      twitterWidgets.async = true;
      twitterWidgets.src = 'http://platform.twitter.com/widgets.js';
      document.getElementsByTagName('head')[0].appendChild(twitterWidgets);
    })();
  </script>


</body>
</html>