Book Inc.: A Software Development Parable

Have you ever wondered what your friends at tech companies do? Imagine the following…

Book Inc. is a company that publishes a novel (piece of software). The company wants to continuously improve their book, so they release a new, updated version every few weeks (agile methodology).  The novel is twenty chapters long, so the company decides to break up the work (because teamwork makes the dream work, right?).

The company has four teams (development teams). Team 1 is in charge of Chapters 1-5, Team 2 is in charge of Chapters 6-10 and so on. The company has the main copy of the book (master branch) in a Word doc on one computer in the company headquarters (code repository). The company e-mails a copy of the word doc to each team’s shared computer (feature branch).

Each team has four writers (developers) and one team leader (project manager). The leader and the writers work together in meetings (backlog grooming) to decide which parts of the book they will fix for this release period (sprint). The project manager, from the team’s shared computer, forwards the copy of the book (local branch) to each of the writers.

The writers spend a couple weeks making edits to the book (development/construction). They have to make sure that they talk to each other so that the team’s chapters make sense. To coordinate, they meet every morning before work (scrum/stand-up meeting) and tell each other what they did yesterday, what they’ll do today, and if they are having any problems figuring out what to write. The team leader will figure out ways to help the writers if they have writer’s block or if they need more information about what to write.

Some teams may feel that their chapters need to be in a certain language (programming language). “It just reads better in the original Italian, of course!” The teams just need to keep in mind what languages the user is able and willing to read. Some languages are better at expressing certain things than other languages, so it may be necessary to use different languages. Usually, it’s easier to just use the same one.

When the writers are ready, they email their work back to the team’s shared computer and work to make sure all of the edits fit together (merging a local branch to a feature branch). The team then prints out the new copy and binds it into a book (creating a build).

They then give the book to a group of editors (QA team) that makes sure everything is spelled correctly. When the editors are happy, they give the book to some special readers from the company who are story telling specialists (user acceptance testers). While the editors checked for grammar, these readers make sure the story will be entertaining to the public. “Why is the sad man sad? Does everyone have to die at the end? Can’t we really just wrap this all up with a dream sequence?”. If there are any changes that need to be made, the writers have to type up the fixes and print out another book.

Once all the editors and readers are happy, the team emails their version of the book to the main computer in the company headquarters. The teams have to coordinate again to make sure all of the edits work together (merging feature branches to the master branch). Then the editors and readers make sure that all of the team’s work makes sense when it’s added all together (integration testing). They’ll also check to make sure if this book is better than the last version of the book (regression testing).

Once everybody is satisfied, the company prints and binds the book from the main computer. They make a lot of copies because there are several book stores (servers) that will sell the book! The team spends all night loading the books into trucks, unloading the books into the store, and making sure they are displayed just right on the shelves (deployment process). The next day, shoppers (users) come to the store to buy the books.

The company has to be very strategic about how the ship out all of the books. In fact, this process (dev ops) is just as important as writing the book! It can be expensive and difficult to print a lot of books and ship to a lot of different stores. It’s easy to ship everything to one store, but what if that store has a flood (server hardware failure)? What if so many people show up to the book store that the book store can’t fit everybody in (failure from high traffic)? It would be a good idea to send the books to a few stores (distributed computing) and have employees (load balancers) that tell shoppers to go to another store if one runs out of books or if another is too crowded.

Meanwhile, the writers all get together to figure out what went well and what didn’t go so well during the past few weeks (sprint retrospective). They write down their lessons learned (knowledge management) and quickly start work on the next version of the book!

The Technical Version of the Story

In the agile methodology, organizations focus on continuous improvement by releasing new features, updates, and bug fixes over the course of a sprint. A sprint is a period of time that includes development, testing, and deployment activities and lasts usually few weeks (or a few days for very advanced organizations).

Development teams create feature branches off of the master branch in the code repository. Individual developers create their own local branches from the feature branch. The team then meets for a backlog grooming meeting to determine what tasks will be completed during the sprint.

Development teams have daily stand-up meetings to ensure accountability and give project managers the ability to remove blocking issues from the team’s path. Once a team has completed their construction phase, they merge the local branches back into the feature branch and create a build of the software. They deploy this build to a QA or testing environment for QA testing. After passing QA testing, business users validate the business logic during user acceptance testing.

Once a build has passed testing, the team merges their feature branch back into the master branch. The testing team will perform integration and regression tests in a staging environment to ensure that all of the teams’ code behaves as expected.

After passing the last round of testing, the team deploys the code from the master branch into production. Organizations follow a set of deployment procedures designed by the dev ops team to ensure that all the code is deployed with the least cost possible. Organizations may also deploy code across different servers and utilize load balances to prepare for scenarios when individual servers are overloaded or fail.

At the end of the sprint, the team has a retrospective to discover how they can improve in the future, and then the process begins all over again.

 

100 Days of Code, Days 19-20: Spreading the Field with Python

For my last few days on the basketball machine learning program, I decided to test different sets of players to see if the program was as accurate as it was with Lebron and Russel Westbrook. I also decided to see if changing sports or comparing the same player from different years had an impact on the program

The results:

• Karl Anthony Towns vs Anthony Davis: 110/111 Correct
• Kyle Lowry vs Dennis Smith Jr: 106/107 Correct
• Tom Brady vs Drew Brees: 18/18 Correct
• 2014 Steph Curry vs 2017 Steph Curry: 34/82 Correct

The algorithm could tell the difference between players even if they were the same position. It could also tell the difference between football players (perfectly, in fact). What it couldn’t do was tell the difference between years for the same player.

I found this very interesting. Despite the fact that Steph Curry has improved over the past few years, his statistics still show the same patterns at a level that confuses the algorithm. Curry has had three years to change and grow and played completely different opponents alongside different teammates (#superteam) but still left the same statistical signature.

What’s next?

It’s been a lot of fun messing around with machine learning, and Python is cool language. For this week, I want to see if the same algorithm can simulate music using today’s pop songs. Come back tomorrow to see how I did it!

 

100 Days of Code, Day 18: Machine Learning for all NBA Players

Using my old basketball machine learning code as a base, I managed to create a machine learning model that can not only take in differently sized datasets but also gives an output of player predictions on a game by game basis. Let’s break down my code so we can see how Python made building this little application a breeze.

Taking in user arguments

The argparse library comes built in with Python. When you add arguments, you can also add the documentation and help messages that users will see if they give an incorrect input. For this program, we are expecting four integers, otherwise we’ll get an error.

1. The row number in the CSV for Player 1’s training data
2. The row number for Player 1’s test data
3. The row number for Player 2’s training data
4. The row number for Player 2’s test data

To run my program with the test data, you’d enter “python main.py 30 74 104 152” into the command line.

Reading in our datasets

The csv library makes it really easy to build readers and writers in Python. This is all we need to enter to populate an array from our data.

Running the machine learning algorithm

Using the user arguments, the program determines what portions of the CSV data are training data and which portions are test data. This is an important distinction for the machine learning method on line 48. I used different arrays to organize the data by player.

Evaluating our predictions

Once we fit our model with the training data, we can predict which player posted the stats in a given game

That’s it.

All it took was 78 lines of code to get this beauty of an output.

Again, this predicted which stats belonged to which player with a 98% accuracy rate (it confused Russel Westbrook for LeBron James once out of 92 different games). Leveraging powerful Python libraries makes this complicated process look easy! If you want to try this out yourself, you can see the newly updated repository on GitHub.

Coming up next…

Tomorrow, I’m going to put my app through its paces by comparing a variety of players. Come back tomorrow to see just how well it will work!

100 Days of Code, Day 17: Machine Learning, LeBron, and Russ Part 2

Last year, I messed around with the machine learning features in the Python library scikit. I created a program that could tell the difference between LeBron James and Russel Westbrook. Using stats from the first half of the season, the program could determine which stats from games in the second half of the season belonged to which player.

That post was pretty popular, so for the next phase of the 100 Days of Code Challenge, I decided to revisit my code to see how I can improve my work. My hope is that after a few days, I will have a readable and reusable program that can tell the difference between any player in the NBA

First of all, why Python?

Other than R and MATLAB, Python is one of the most used programming languages for academic research. It allows users to easily manipulate arrays of data and provides some great open source libraries for mathematical analysis (numpy and scipy, for example). Plus, Python is really readable.

What does the current program do?

The current iteration of the program takes in data from over 20 different statistical categories for both players for the first 40 games of the 2016 season. This data is used to train the model. Training a model means that we give our machine learning algorithm enough data points to identify future inputs based on specific characteristics.

Using the trained model, the program compares the stats from the rest of the games to a profile it’s developed for each player. It correctly identified each player with 98% accuracy.

Next steps…

Check out the newest repository an GitHub so you can run the program locally. I’m still working through some issues with how I built my arrays (I’ve come a long way since 2017), but this will be the basis for the coming days!

For tomorrow, I will focus on repairing the arrays and abstracting the code to accept data from different players. Check back tomorrow to see my progress!

100 Days of Code, Day 16: Lessons Learned so Far

Today I wrapped up development on integrating the Angular app with the GoSQL database. It’s been a long time coming on this one! Everything is now on GitHub if you’d like to see the code. After finishing three small projects in three different languages, there’s a few things I’ve learned that I wanted to share.

#1: Good architecture is important in projects of every size

You all saw how tightly coupling the application and the database caused a lot of problems. And that was in very small application. Spending some time at the beginning of a project to map out the potential interactions between parts of the system will go a long way to prevent issues down the road.

On the flip-side, I spent some effort making sure that the GoSQL database API was reusable. Next time I need an instance of a GoSQL database, I won’t have to spend a lot of time configuring the data access layer. Thinking a head now will save development time later.

#2: Version control is your friend

I started using Git as a way to share my code with all of you.  I quickly learned that version control has some other useful benefits. When I was developing the Angular app and realized it would be too difficult to force JavaScript to make TCP/IP calls, I had already made a lot of serious changes to my code. Using Git, I was able to pull a fresh copy of the code before I made my changes and quickly pivot to HTTP. It would have taken hours to remove all of the new dependencies and changes I had introduced without version control!

There was a situation where I took a different laptop with me on the way to Houston. Using Git, I could pull down all of my current progress and start working immediately! These are just some of the ways a good version control system can make your life a lot easier.

#3: Get outside of your comfort zone

For the challenge so far, I’ve built one app in a language I already know and two in languages that are new to me (Go, JavaScript/TypeScript). While it was nice to get some repetition in .NET, learning languages that are a) net new to me and b) functional when I’m used to object oriented has forced me to grow a lot in the past two weeks. It might have been easier to learn a language like Java that is similar to C#, but learning the difference in functional languages vs object oriented has given me a better understanding of computer science and software development in general.

I’d recommend that you also get outside of comfort zone. If you are used to Javascript and Node.js, learn an object oriented language like Java! If you are used to open source stacks like Ruby or Python, spend some time in a larger enterprise-style framework like .NET. Pushing yourself may be painful and unfamiliar, but it’s the only way to learn!

Coming up next…

Check in tomorrow for the start of a new project!

100 Days of Code, Days 11-15: WebAPI to the Rescue!

This weekend, I focused on solving the integration issues between my GoSQL database and my Cost Calculator app. To quickly summarize, I was having ptoblems making a TCP/IP call from the app while it was running in the browser as well as with how the app would handle returned data.

The solution…

As mentioned in my post on Day 10, JavaScript wasn’t really built to use TCP/IP. It can, however, make HTTP calls very easily. To keep from putting a square peg in a round hole, I decided to create an API to serve as my data access layer between the database and the app. The app will make HTTP calls to this API to retrieve and save data.

This solution not only uses the preferred transmission protocol for JavaScript. Having a data access layer that can properly format requests for data before sending the call onto the database makes performance more consistent. Besides, giving a client app direct access to your database can only cause problems later on down the road.

Choosing a Framework

I work with API’s at work a lot at work, almost exclusively in C#. Since this was the framework I’m most comfortable with, I decided to create my new data access layer using Microsoft’s WebAPI framework. However, fear not, open source friends! I used .NET Core so that everybody could still play around with the code. You can build and run the API from the command line, just like the other apps I’ve developed during this challenge.

Under the Hood

The API controller has four methods to line line up with each of the database commands: GET, POST, PUT, and DELETE. The app just has to call one of these methods and provide the id and (in the case of POST and PUT) the desired value. Each controller method calls a helper method that communicates with the database.

The meat of the helper method is listed above. As you can see, I’m trying to stick to the time-tested practice that every method should just “Do One Thing”. I’ve created different methods to create the socket for the TCP/IP call, establish a connection with the database, and transmit the command.

I’ve also added error handling to return a 500 error and a helpful message if something goes wrong connecting to the database. If I had left the app and the database tightly coupled, any errors would have been hidden away in the developer tools, creating frustration for the user.

Coming up next…

The last step is to finish testing how all of the apps play together and then upload everything into GitHub. Check back in tomorrow to see how it goes!

100 Days of Code, Day 10: With Integrations, the Devil is in the Details

Well, today I have hit a snag. Integrating the GoSQL database with the Cost Calculator app is proving to be a challenge. The 100 Days of Code Challenge only dictates that I spend an hour a day on coding outside of work. Unfortunately, an hour is usually all the time that I have. There are a couple architectural issues that I need to solve, and fixing these problems will take several days.

There is a silver lining here, however: I know I have the opportunity to take you all on the journey with me as I attempt to integrate one service with another. With architectural patterns like micro-services and distributed computing coming to the forefront, it becomes incredibly important to have a mental of modal of how to approach integrations

Problem #1: Communicating Between Apps

The HTTP protocol (redundant, I know; it’ll happen a lot so bare with me) is one method of communication between applications over the internet. JavaScript can make these types of calls really easily. My app will run on a single server/computer, so I want to use the TCP/IP protocol to communicate over sockets. Unfortunately, there is no native way in JavasScript to make TCP calls out of a browser, the exact place the app will be running.

To solve this issue, I will be using a third party tool called WebTCP to facilitate TCP calls from the browser. This is another integration that I will have to be careful with as well! In your application, make sure that the system can handle as many edge cases as possible when it comes to communicating with an integrated services. Be prepared to handle lost calls, incorrect calls, or a sudden, large influx of calls. It doesn’t matter how good the code is within the service if the individual services can’t talk to each other.

Problem #2: Returned Data Format is Uncertain

Here is how my data service looks after an hour of research, integration, and coding. See an issue here? As mentioned in earlier posts, the GoSQL database that I’ll host on port 3333 is schema-less. Why is that a problem? Unlike a relational database, my data could come back in any form. When I try to push the data into the resources array, it may be incompatible with my resources object.

In your application, have skepticism about what a service can return. Be prepared to handle any data that your service can throw at you, because eventually there will come a time when you ask for a specifically formatted JSON object and get a orangutan, fruit bat, or breakfast cereal in return.

To summarize…

Integration points are perfect areas for failure. Networking concerns and unreliable payload formats can take down your app just as well as a poorly written line of code. The rest of this week will be spent addressing these concerns. Stay tuned for the results!

 

100 Days of Code, Day 9: Child and Parent Components in Angular

Day 9 was spent on final edits to the components in the Cost Calculator Angular app from Days 4-8. A big part of making this app more modular and organized was breaking down the app into child and parent components.

Why would you do that?

Decoupling portions of the code means that changes only have to happen in one place. This also insulates an errors that might occur into a single component. For example, in my code above, if I had an issue with my Cost Total Detail module, the rest of the parent component could still load. If all of the code was contained in my component and a change caused an error, the whole thing would break and fail to display.

Ok, I’m sold. How would you do that?

In the code snippet above, we are injecting an object from the parent component (in this case, the Cost Total) into the child component (the Cost Total Detail). Cost Total Detail is considered a child component because it is expecting a Cost Detail from the parent. I used the Input modifier in the Cost Total Detail to configure the component to receive the object.

This happens in several places throughout the app. You can check the code here and see that Resource has a child called Resource Detail, and Cost Total Detail even has a child called Work Item. Together, the components work together to build the following.

What’s next?

Tomorrow I begin to update the data service to actually make calls to the GoSQL database. Check back tomorrow to hear all about it!

 

100 Days of Code, Days 4-8: Keeping Track of Your Costs with Angular

The wait is over! After a rewarding time rebuilding a house in Houston with the Credera team, I’m back and ready to give you some insights into what I’ve been coding over the last four days. I know everyone was on the edge of their seats all weekend preparing to learn what I’ve been up to!

I spent Days 4-8 of my #100DaysOfCode challenge building out a cost calculator in Angular that will use my GoSQL database from Day 3. While I’m not quite done, I’ve learned a lot about how the framework works and how to architect an application in Angular. This post will focus specifically on the front-end development. We will get into adding the GoSQL database later this week.

What is Angular?

Angular is a front-end Javascript framework maintained by Google that was released in 2010. Angular is known for its improved data binding and Model-View-Controller structure that gave it an advantage over older Javscript frameworks like jQuery.

So what is this app?

I am creating an application that calculates total cost for hourly resources. The user will have the ability to enter a name and cost per hour for a resource as well as give the number of hours worked by a resource for a given client. This will give a total for that client. Using the data binding in Angular, when a user updates the cost of a resource, it should also update the total for the client as well.

Getting started in Angular

It’s easy to get started building an Angular app. Once you download the Angular Command Line Interface (CLI), you simply have to type “ng new [Project Name]” into PowerShell, Terminal, or Command Line. This command will create a fully scaffolded project that you can run locally using the “ng serve” command. It’s really that simple!

The building blocks of an Angular App

The entire app is made of components. Angular components contain a view (made with HTML), styling (made with CSS), a controller (made with TypeScript), and a “specs” file to hold some settings (also made with TypeScript). You can create a scaffolded component using the “ng generate component [Name]”.

You control the data displayed in your view by inserting angular commands into the HTML. The Angular commands are defined in the controller and can load updated data in real time. As mentioned earlier, this is Angular’s advantage over older frameworks or general front-end JavaScript. In my app, updating information in the resource details text box will instantaneously change the data displayed in the resource box above. It is my goal to make sure changes on the resource component will update data in the cost total component.

Services at your service

Angular apps also contain services, which are importable functions that can be used throughout the program. In my app, I have a data access service that will be used to access the GoSQL database. Right now, it is returning the dummy data you see above. Services are injected into the component’s controller using dependency injection.

Next steps

My goals for this week is to finish up wiring the app together so that manipulating the resource cost updates the total. I’ll then update the service to read and write to an instance of the GoSQL database. Check my progress and play around with the code on GitHub and be on the lookout for tomorrow’s blog post!

100 Days of Code, Day 3: It’s Alive!

After 3 days, I finally have working code! My NoSQL database (affectionately known as GoSQL) can receive database commands and properly manipulate data. It may not be pretty but it’s still really rewarding to have a “finished” product in programming language that I had no knowledge of on Monday.

Finishing the project required adding the ability to read and write from and to a JSON file. I’ll show you how to add this into your Go code and one trick that might be blocking you.

Reading from the JSON Database

To read our file, we use the ioutil package. We can take the result and use the Unmarshal function. Unmarshal will deserialize the JSON from a byte array into a structure that you define. In this case, it will deserialize into our db variable of type Database

Writing into the JSON Database

Once we’ve manipulated or data, we want to save into our JSON file. We will use the sibling method of Unmarshal, Marshal!

This time, we are creating a byte array using JSON encoding that we can write to our file.

An important note!

If you are going to serialize a struct to JSON, the struct’s properties have to be exportable. This is notated by capitalizing first letter of a property’s name in the struct definition. Making a property, variable, constant, or struct exportable means that its usable in other files. Think of this like using the “Public” or “Global” keyword in other programming languages.

What’s Next?

I now have a database that can create, update, get, and delete key value pairs and store them in a file for later use. It’s rather rudimentary, but it works! This opens up opportunities for other apps that need to store simple amounts of data. Come back tomorrow to see how we can put this database into action!