Posted: July 2nd, 2010 | Author: Julio Hernandez-Miyares | Filed under: Cloud Computing |
Spent a few quality hours on Wednesday evening July 1st at a Meetup sponsored by Pivotal Labs to discuss their Tracker Project Management tool.
Quintessential New York style just off of Fifth Avenue on 27th Street in a Class B or perhaps C building that probably served as a garment manufacturing
facility back in the day when those streets were chock full of hand driven garment tracks.
Empire State Building
No, this is not quite the view from the 18th floor of their loft space as I was there to work and learn and not act as a photographer but in fact their view was better with the Chrysler Building and the rest of Gotham glistening as the sun set over the Hudson River. No office space surrounded by quadrangle seas of green in this City unless you include the very public Central Park and the other architecturally renowned parks sprinkled throughout the boroughs.
Pivotal Labs is a software company originating in San Francisco with a major outpost here in the Big Apple with a focus on providing Technical Software Engineering services to clients. Think of like an Agency but with Developers. That is not how I came to know them though. As Software Engineers, they architected and engineered for their own use an internal Project Management Tool to manage their Agile development process and eventually made it available as what I would characterize a Cloud Service. What can be better then that as a start? A project Management tool where the initial customers for the feature set were developers. Though I have no way of verifying , they did say they have to date about 100k users and they have at least one because we at Jittr use it.
What is Tracker? First don’t call it Pivotal Tracker as I did , it is just plan Tracker as the principals like to reinforce.
Well if you are conversant with Agile Story definition and management tools like Rally that provide a fundamental utility to manage and coordinate Agile Development in all it’s phases, that is what it is.
You define a story , describe it, and leave it in the “ICEBOX” which is a metaphor I find attractive. You have a planning session with the team and here a team is small (6 or at most 10) move the estimated in terms of amount of work accepted stories into the Backlog. It uses simple estimation based on either 1,2 and 3 or fibonaci sequence. The fact is they are relative measures. You don’t estimate so much like “this widget will take 23 hours” to build
This product is apt for those shops that want something powerful because of it’s simplicity and focus, cheap (we use the free version because of our current small size), and who want to be able to influence an open group of developers in furthering the efficacy of the existing product.
Who is it not for? Well, if you have money to burn and like paying license fees for the fine dinners or box row seats you may garner by signing a wide expensive perpetual site license. If you are so process driven that without integration with every conceivable project management package or if you need specificity in the estimation activity bordering on creating mirages that you can present as verifiable fact because the tool say’s so.
On the other extreme if anything more disciplined then redacted napkins with the feature descriptions or endless email threads with point -counter point to describe a feature, then the tool is not for you.
Posted: June 29th, 2010 | Author: Julio Hernandez-Miyares | Filed under: Cloud Computing |
Enhanced Web Robots will eventually enable us to
finally get to the life of learning and leisure that computers have so far failed to do. What kind of Robots or Bots as they are lovingly known are we talking about?
Well , you give a natural language command say “what are the food handling and preparation regulations in New York’s Columbia County and what forms and inspections are necessary
to get a certificate that allows me to prepare and sell on outdoor food carts..”.
This is a real life problem I am currently working the old fashioned and best manor today via Google Searches for my Cuban Cuisine Catering company I recently formed.
Thank Goodness for Google to help me find the answers to what is a problem I don’t even knew how to solve at the outset other then the fact I knew there are numerous competing agencies and regulations in the food growing, food preparation and sale business especially when dealing with perishables like dairy and meats.
Why not strive to solve this difficult problem even though it won’t be solved in a quarter or even a few years. At least these Robots won’t have to have vision and those normally physical issues that need to be
solved so I can have a robot that can be my house maid.
Posted: June 23rd, 2010 | Author: Julio Hernandez-Miyares | Filed under: Cloud Computing, Mobile, Technology, Technology Outsourcing |
As Jittr’s focus are Smartphone applications , primarily Android but eventually and soon iphone, we are caught in the dilemna of how to test on a cross section of the handset devices we are targeting.
Test Engineering is one of those technical areas that we wish to outsource to a team in a far away place where supply of engineers is high, competence equally high but costs are way low. India is one such place, though not the only one but since my principals and myself have years of experience with teams in India, that is the out-source geography we are most familiar and comfortable with.
So far so good but we are not talking about testing on an emulator running on Windows or Mac workstation. That is good for development testing. Real test engineering has to test on the actual devices running the actual variant of the Android OS possibly customized by the Wireless Carrier.
Device Anywhere to the rescue!
I just completed the live webinar on DeviceAnyWhere Test Center application. Quite impressive! As an aside, I signed up for a free open trial prior to signing up for and viewing the howTo seminars and put the product (both the Test Center and Test Automation Facility) through it’s paces but the Web based live walkthrough was still worth the hour of investment of my time. This is a profitable (or so they say) startup in Cloud Services with about 2,000 clients and their main claim to fame is making handsets available via the cloud to engineering teams that need to test their mobile applications on as well rounded set of different handsets as possible. They have multiple data centers including one in Europe, the USA of course and a new one opening in Brazil shortly. The data centers are centers where the actual physical devices are located. All testing is done on real handsets which a tester acquires via the Test Center/Automation application prior to running through the test cases.
All activity performed on the physical unit is streamed in a pixel perfect manor to the testers’ workstation which renders in an image which resembles the physical device.
Also, those pixels are captured to form a basis of screenshots that show the visual state of the device as the testing proceeds.
The Test Center also includes it’s own feature to define and manage test cases in a centralized manor. No more having to use Excel or something similar for the same purpose.
The probable use case would be for someone in Jittr, let’s assume myself for the moment, would define the testcases within the test center and then outsource the actual testing to a remote team in India. The Test Center makes this easier to accomplish. The Test Case Manager has the functionality to assign pass or fail to the test, attach screenshots of what was observed on the handset for each particular test
Another useful feature is the ability to share an acquired handset to someone to put the application through it’s paces.This feature I have not exercised yet so I am not sure if the invitee has to download the testcenter application to actually avail oneself of this feature. I find this a potentially useful way to let someone , say one of my principal none technical colleagues, someone like a Product Manager who is remote, run the current state of the application through a real device even if the real device is somewhere hidden out in the clouds. A lot simpler I hope and will confirm then having them install Eclipse and the Android development kit to have the use of an emulator to simulate the functioning of the Android Application.
Overall, appears quite impressive from my kicking the tires and the Webinars and as the world moves quickly in the mobile market, this company as well as others should form the bedrock of the way mobile testing is done except for the absolutely largest companies.
I am in the process of converting from a free trial account to a paid account. They have flexible plans for startups like Jittr where breaking the bank is not an option. I will come back to this post with information as I put it through it’s most important paces which is for real and with a virtual team using it as the foundation for test engineering.
Posted: May 1st, 2010 | Author: Julio Hernandez-Miyares | Filed under: Cloud Computing |
What will they think up next that can be put on the “Cloud”.? Well that is tongue and cheek in in a way. Probably any software service can be adapted to form a service in the Cloud. The Cloud simply defined as something somewhere obviating me of worrying about it’s operation and the only interest and concerns are
- it’s reliability (there when I need it)
- scalability (ramps up with my needs)
- semblance of privacy controls (only those that I grant access get it).
Last week the individual writing up the product requirements for the Social Gaming Mobile app that we are working on at Jittr.com suggested we consider using Pivotal Tracker for managing requirements. Pivotal Tracker is an online tool to manage the Agile methodology with the normal adherence to Stories, Backlogs, Current Sprint Log for accepted and inflight work and the sundry reports that attempt to depict software development productivity such as Velocity ie how many story points are completed in a given sprint development cycle.
It also has an REST based API specification which I have not had the opportunity to try out that allows exercising it’s capabilities and the project information contained therein increasing the flexibility exponentially if the need to integrate with other project based tools arises.
As a very small boot strapping start-up, we are not interested in using this tool to the extent a large company with more defined processes and much larger team and reams of executive or program management who spend their lives monitoring how productive their technology development teams are operating. Nevertheless it’s core capabilities to define stories and share in a centrally accessible location and allow updates whether on the product or development perspective and all for the grand sum of $0 makes it a useful tool in our arsenal to communicate and execute quickly.
After a few weeks, will update this post not only with additional reflections on the quality and efficacy of this tool but also the extent it makes sense to use the discipline of Agile for a startup fitting the profile of Jittr.
Posted: April 10th, 2010 | Author: Julio Hernandez-Miyares | Filed under: Amazon Web Services, Cloud Computing |
Needing a search engine for a current under-development Mobile Application, I decided to follow the route of an Amazon Machine Image (AMI) running SOLR which is a version of Lucene with a nice XML over HTTP interface. This is part 1 of a multi part post as I meander my way to a truly production ready , fully stacked Search engine capability including the persistent data stores that will feed the Search Engine indexing operations. A good reason for this blog is solely as documentation for what I have done to get there. There is plenty of documentation surrounding Amazon Web Services including Amazon’s official documentation but at least for me , it all appears disjointed at times, dated and not correct at least as the current apis and services work and getting something done for a “newbie” and I am stretching the definition of that term can often times be grueling or at least more time consuming then it has to or should be.
The first part of the series of posts will have as it’s conclusion an instance of SOLR powered by the version of Jetty that comes with the SOLR build. I will follow up with a Tomcat powered version of SOLR in part 2 or 3 of the post. For now, just wanted to make the exercise more about getting you own functional AMI instance configured, defined, registered and launchable.
So here goes nothing;
First, my JittrSolr (Jittr is my company) AMI instance I built was initially based on the Fedora Core 8 (AMI Id: ami-b232d0db)
Minimal Fedora Core 8, 32-bit architecture, Apache 2.0, and Amazon EC2 AMI Tools.
First I downloaded the two packages I will need to add to the base Fedora Machine Instance to my local machine. They are
jre-6u19-linux-i586-rpm.bin for java and apache-solr-1.3.0.tar for SOLR.
*NOTE – I could have saved the download and upload cycle by just using “wget” from the command line of the Fedora Machine but I did not have the exact url for the download package. That will be for a future exercise. It is a time saver of sorts because the upload to the Amazon Machine Instance was time-consuming. We are talking minutes not hours but during those minutes you are simply waiting and 12 cents an hour, though you are not going to break the bank , every minute does count.
Firing up a Machine Instance through the AWS console and then obtaining the secure shell commandline string to connect to the machine instance
ssh -i imac.pem root@ec2-174-129-66-51.compute-1.amazonaws.com
I am ready for the first part of uploading the two packages to the Machine Instance.
*NOTES – the ssh command string will change depending on the machine domain you have been allotted and the actual key pair you assigned to the instance when you were instantiating it. For convenience and since I don’t need anything else, I just use the native terminal program that comes with Mac Snow Leopard.
Also, make sure the security group you choose when configuring the Machine images has the port 8983 open for http. This is the port SOLR uses to listen for search requests.
I use secure copy (scp) from my local machine’s terminal session to upload the two aforementioned packages.
scp -i ~/Desktop/ec2/imac.pem ~/Desktop/ec2/packages/java/jre-6u19-linux-i586-rpm.bin root@ec2ec2-174-129-66-51.compute-1.amazonaws.com:/root
scp -i ~/Desktop/ec2/imac.pem ~/Desktop/ec2/packages/solr/apache-solr-1.3.0.tar root@ec2ec2-174-129-66-51.compute-1.amazonaws.com:/root
Both packages are uploaded to the /root directory of the Machine Instance. Connecting to the Machine Instance using the ssh string detailed above and performing a little housekeeping from the machine instance /root directory
To validate I have the stock , simple SOLR working
-
cd /usr/local/apache-solr-1.3.0/examples
- java -jar start.jar
to start up the SOLR instance
- Entering “http://ec2174-129-66-51.compute-1.amazonaws.com:8983/solr/”
within a browser should now give you the “Welcome to Solr” page with a link to the admin panel of Solr.
Note – as stated above you must have port 8993 for http open in your security group for this to work.
From the terminal session of the machine instance, now comes time to package your configuration and system software and then register it as a private machine instance.
-
ec2-bundle-vol -d /tmp -k /root/pk-???.pem -u [Your AWS AccountID] -s 2048 -c /root/cert-???.pem
Will bundle the machine image into the /tmp directory. I did have to upload both my private key and certificate. Various documentation stated I did not need the certificate but it would not work without it. Also, though I used the -s option for the image size, some documentation I reviewed stated it was best to leave it to ec2 to decide. There was a lot of trial and error. It worked with -s 2048. Also , the -u option is your AWS Account ID which you can view if you have not memorized it from you AWS Account Web page. I removed the “-”’s but I did see comments where people having problems getting the command to work fixed the problems by reintroducing the “-”’s. It worked for me with no dashes.
-
ec2-upload-bundle -b jittrsolr -m /tmp/image.manifest.xml -a YOUR_ACCESS_KEY -s YOUR_SECRET_ACCESS_KEY
Will upload the machine image to Amazon’s S3 in the bucket defined by the -b argument. The bucket will be created if it doesn’t exist.
the -a and -s arguments are for your Access Key and Secret Access key respectively which are also available from your AWS Account web page in the Access Credentials section.
-
ec2-register jittrsolr/image.manifest.xml -n jittrsolr_small
Finally to register your new Machine Image executed from your local terminal session , not the Machine Instance terminal sessiopn
-
ec2-describe-images -o self
If all as gone as planned, you should see your new private instance listed.
It will also be listed on the AWS EC2 Console in the Private AMI tab
Posted: March 1st, 2010 | Author: julio.luis.miyares | Filed under: Cloud Computing, Cloud Map Reduce |
If understanding Data is the true measure and worth of collecting it in the first place, what is now standing in the way at a minimum of the capability to crunch the data using a slew of tools adapted for that purpose? The answer is nothing , well kind of. With the publicly available services in the Cloud such as Amazon’s MapReduce, the inherent capability it there for all but the most uninitiated to make sense of the mountains of data point collected by a typical high volume website.
Of course one can just plugin Google Analytics onto their website and they are good to go with a robust solution that captures, aggregates ,slices and dices and presents pretty reports via a web form about a host of activity on a website. Nevertheless there are always instances where the need to unlock additional understanding inherent in all that data requires some additional processing and some additional tools.
One of the biggest costs for large scale data processing has usually been the need to have an operational infrastructure in place to excise, collate, aggregate and slice the mountains of data that are generated by your typical website or application usually measured in gigabytes and terabytes. Of course all that operational infrastructure is always bound to some strategic and business critical processing leaving precious little horsepower available for for those extraneous random questions that come up on a recurring basis for any business. By the time the request for that question to be answered is queued up and delivered by the typical IT group ,it is highly likely someone more nimble somewhere else has already answered the question and gotten a lead on adapting their business with the answer in hand.
With the wide availability of Cloud Services such as Amazon’s Map Reduce, the timeline to answer the question can be markedly reduced. Admittedly you still need Technologists either in house or outsourced that understand the components of Map Reduce and how to actually leverage it’s power for the needs of the business but the need to wait for precious cycles on the IT’s host complex, partnering Software engineers and Operational Admins to make the environment available with the hosts and necessary software and connectivity to actually run jobs can be short circuited to almost zero time.
You also need the Analysts that can come up with pertinent questions specific to the business they are in and with an understanding of the data elements and relationships between them that are currently captured or can be captured and can help answer specific questions. The technology is not smart enough to do it by itself.
A non technical question , are mature companies with a long standing streak of being in business availing themselves of this form of Cloud services computing? I would not expect them to have the same drive to operate on a shoestring as the majority of startups do but are Cloud services of viewed as an valid option by most IT groups in mature organizations?
Yes, more then a handful of companies outsourced their Development but especially Networking over the course of the past few decades leaving those esoteric functions to entities like MCI that had scale and a knack for staying on top of the evolving technologies shuffling bits around.
Posted: February 26th, 2010 | Author: Julio Hernandez-Miyares | Filed under: Cloud Computing |
At least in the technical realm, the noise around cloud computing has become a veritable roar with all of the usual hype associated with the promise of alleviating some of the more difficult aspects of operationalizing the assorted services that constitute the web as we know it. In many respects the hype is legitimate and is making technologists’ potential to concentrate on the core feature set and leave the operations to someone else that much greater.
It the context of this post I will differentiate between Cloud Applications like Flickr or Delicious and countless others that provide product functionality directly to a consumer requiring precious little if any technical acumen and Cloud computing services like Amazon Web Services that provide a set of fundamental building blocks to build products.
Before I move on, a Cloud application I have recently started using that leverages Cloud services namely Amazon’s is Backupify. As the name implies, it backups various well known applications such as Flick, Delicious, wordpress blogs etc and the backups are persisted via Amazon’s Web Services. So here is an Cloud application that uses Cloud services to flesh out it’s features. The power of this model of crafting products is enormous. The time to market is considerably reduced as the purveyor of services can quickly spin up a set of functionality without much regard to establishing a data center/rack space and all that is involved in operationalizing something of this sort. Also, the vendor of the service is able to manage costs and attendant risk during the hectic startup period. They generally only have to pay for as much service as they consume which is governed by the adoption rate of new customers. This makes the operational costs almost entirely variable instead of fixed which provides the necessary risk mitigation especially for new startup like businesses. If the adoption rate is low and the service does not win out in the marketplace, there is no need to sell equipment at fire side sale prices to wind down an unsuccessful business. Conversely, growth can be managed in a more balanced manor by only tapping into the services needed. Also, for those particular instances where one has to deal with spikes in usage , the need to have an operational footprint large enough to support the anticipated spike ends up revolving around tapping into extra services from the Cloud to support the spike and when the spike is over, the horsepower is made available to whomever else out in the cloud when need the capacity.
This makes history of the scenario where one would need to manage fixed operational infrastructure sufficient to deal with the highest volume of usage but where that infrastructure is often left under-utilized save for the few instance of spikes.
