The Arduino IDE 1.5.3 introduced some new features, such as support latest board Yun, or introduces new libraries and samples. But I found it is too buggy to use.
1. Compile code against Arduino Nano fails due to parameter mcu passed to avrdude is missing.
The issue seems caused because the new IDE merged the menu items for Nano board. But for some reason, the configuration haven’t been updated accordingly.
Choose Arduino Duemilanove and Diecimila instead of Arduino Nano. Nano uses same chip as Duemilanove(ATmega328) and Diecimila(ATmega168), but uses a different PCB design. So the binary should be compatible.
One of the most annoying thing of Mac is that encoding, espeically you’re living in a non-Mac world.
Mac uses UTF-8 as the default encoding for text files. But Windows uses local encoding, so it changes according to your OS language. For Chinese users, Windows uses GB2312 as the default encoding.
So usually the movie subtitle files, the song lyrics files, the plain text novel files, the code contains Chinese, which you downloaded form web sites or recieved from others usually cannot be read because of the wrong encoding.
So I really wish to have an item in my finder’s context menu that I can adjust the encoding of selected files with 1-click.
Luckily, with the help of Ruby, Automator workflow and Mac OSX service, it isn’t that hard.
So basically, OSX loads all the workflow files saved in ~/Library/Services/, which is displayed as Context Menu in finder.
To build the service, work through the following steps:
1. To create a new service, just pick Service in Automator’s ‘create new document’ dialog.
2. Set service input as “files and folders from any application”.
3. Run Ruby Script to transcode the files
Add a “Run Shell Script” action to execute the following ruby code, which is used to transcode the files passed to service. ( For more detail about how to embed Ruby in workflow, check out Using RVMed Ruby in Mac Automator Workflow )
Make sure the input is passed as arugments to the ruby script.
4. Display a growl message when processing is done
5. Prompt user whether to keep the backup files
I use a Ask for confirmation action to ask whether user want to keep the backup files.
The workflow will abort if user clicks “No”, make sure you updated the text on the buttons, and texts are put on right button.
6. Add script to remove backup files
Add another “Run Shell Script” aciton to execute another piece of ruby code.
7. Display notification to tell user that backup files has been deleted
TIP: The transcode ruby script requires Ruby 1.9+, but Mac OS X default provides Ruby 1.8.3, which doesn’t support encoding. To interprets workflow embedded code with ruby 1.9+, please refers to Using RVMed Ruby in Mac Automator Workflow
HINT This content is obsleted on OSX 10.9 Mavericks
Embed Ruby into Automator Workflow
Automator workflow has the ability to execute ruby code, but it is not that obvious if you doesn’t know it.
To embed ruby code into workflow, you need to create a “Run Shell Script” action first, then choose “/usr/bin/ruby” as the shell. Then the script in the text box will be parsed and executed as ruby code.
So, from now on, you know how to embed ruby into automator workflow.
Use RVM ruby instead of System Ruby
By default, Automator will load system ruby at /usr/bin/ruby, which is ruby v1.8.7 without bundler support. For most ruby developers, they must have installed some kind of ruby version manager, such as RVM or rbenv. As to me, I uses RVM. So I wish I could use RVMed versions of Ruby rather than the system ruby, could be ruby 1.9.3 or even ruby 2.0 with bundler support.
To use the RVMed ruby, I tried several approaches by hacking different configurations or files. And at last, I made it doing this:
RVM provides a ruby executable file at ~/.rvm/bin/ruby. On the other hand, /usr/bin/ruby is actually a symbol link that pointed to ‘/System/Library/Frameworks/Ruby.framework/Versions/Current/usr/bin/ruby’.
So what we need to do is to replace the the symbol link with a new one pointed to ~/.rvm/bin/ruby.
I found a very interesting phenomenon in Java type reflection when building an Android app.
I’m trying to build a common mechanism to serialize Enum into integer when writing it into database. To make it more flexible so I fetch the value type dynamically by using reflection. So I have the following code to check whether the value to be written is an enumeration:
I built the converters into a responsibility chain. The converter is applied only when match method returns true.
In the converter, I check the type with isEnum method. I expects the method yields true when the value is a enumeration. But later I found it doesn’t work as expected. And the behavior of this method is really confusing!
So the value FULL has a different type than ServiceStatus as I expected, the type of FULL is actually a subclass of ServiceStatus. And the enumeration value FULL is a singleton instance of the anonymous sub-class.
The the most unexpected behavior is that the isEnum method only returns true on Enumeration class itself, not its subclass!
To resolve this issue gracefully, I changed my implementation a little bit. Here is the updated implementation:
I’m working on an Android application recently. During the development, our new app inherited a legacy SQLite database that created by a WinCE application.
Android announce that it supports the SQLite database, so we don’t worry about that too much. But when we try to open the legacy database with SQLiteDatabase.openDatabase, it throws exception on our face!
After debugging, we found to open SQLite database with SQLiteDatabase class requires the database to have a special table called android_metadata, which contains the locale information. And SQLiteDatabase yields exception when the table cannot be found.
The table will be created automatically when SQLiteDatabase creates the database, but if the app need to open the database not created by android, some patch is needed.
Have been on travel once again, and really found filling the ThoughtWorks T&E expense form is really painful.
So I spent some time to refine the scripts that wrote last year when I was on Travel, which could be stimulate the pain when filing the form.
This is the 2nd post of the Node over Express series (the previous one is Configuration). In this post, I’d like to discuss a famous pain point in Node.js.
There is well known Lisp joke:
A top hacker successfully stole the last 100 lines of a top secret program from the Pentagon. Because the program is written in Lisp, so the stolen code is just close brackets.
It is a joke that there are too many brackets in Lisp. In Node.js there is a similar issue that there are too many require. Open any node.js file, usually one could find several lines of require.
Due to the node’s sandbox model, the developer has to require resources time and time again in every files. It is not so exciting to write or read lines of meaningless require. And the worst, it could be a nightmare once a developer wishes to replace some library with another.
“Require hell” isn’t only for node.js, but also for Ruby apps. Rails has solved it gracefully, and the developer barely needs to require anything manually in Rails.
There are 2 kinds of dependencies in rails app, one is the external resource, another is the internal resource.
External resources are classes encapsulated in ruby gems. In ruby application, developer describe the dependencies in Gemfile, and load them with Bundler. Some frameworks have already integrated with Bundler, such as Rails. When using them, developer doesn’t need to do anything manually, all the dependencies are required automatically. For others, use bundle execute to create the ruby runtime with all gems required.
Internal Resources are the classes declared in the app, they could be models, the services or the controllers. Rails uses Railtie to require them automatically. The resource is loaded the first time it is used, the requiring process is “Lazy”. (In fact, this description isn’t that precise because Rails behaves differently in production environment. It loads all the classes during the launching for performance reason).
Autoload in Node.js
Rails avoids the “require-hell” with two “autoload” mechanisms. Although there are still debates about whether autoload is good or not. But at least, autoload frees the developer from the dull dependency management and increases the productivity of developers. Developers love autoload in most cases.
So to avoid “require-hell” in Node.js, I prefers autoload mechanism. But because there are significant differences in type system between Node.js and Ruby, we cannot copy the mechanism from ruby to node as is. Therefore before diving into the solution, we need to understand the differences first.
Node.js Module System
There are a number of similarities between Node.js and ruby; things in node.js usually have the equivalences in ruby. For example, package in node is similar to the gem in Ruby, npm equals to Gem and Bundler, package.json takes the responsibility of Gemfile and Gemfile.lock. The similarity enables porting autoload from ruby to node.
In some aspect, there are similarities between Node.js and Ruby, but there are also significant differences between them in some other aspects. One of the major differences is the type system and module sandbox in Node.js, which works in a quite different way to Ruby type system.
In ruby, it is a lot better. With the help of the well designed type system, types are shared all over the runtime, a developer just needs to require the types not yet loaded.
In fact, with the help of JSLint, CoffeScript and some other tools, developers can avoid global leak easily. And global sharing isn’t the source of evil. If abuse is avoided, I believes a reasonable level of global sharing could be useful and helpful. Actually Node.js have built-in a global sharing mechanism.
To share values across file, a special variable global is needed, which could be accessed in every file, and the value of which is also shared across files.
Besides sharing value around, global has another important feature: node treats global as default context, whose child you can refer to without explicitly identifying. So SomeType === global.SomeType.
With the help of global, we find a way to share types across files naturally.
Rails’ autoload mechanism loads the classes lazily. It only loads the class when it is used for first time. It is a neat feature, and Rails achieve it by tracking the exception of “Uninitialized Constant”. To implement similar feature in Node.js, tracking exception is hardly feasible, so I choose a different approach, I use Property.
With property, we make the autoload lazy!
To make autoload work, we need to create a magic host object to hold the type variables. In my implementation, I call the magic object Autoloader
we need to require a bootstrap script when the app starts, which is used to describe which types and how they should be required.
In the initEnvironment.coffee script, there are 2 very important classes that are used:
* AutoLoader: The class that works as the type variable hosts. All the magic happens here.
* PathHelper: The class used to handle the path combination issue.
The code above are part of the Express over Node, to access the complete codebase, please check out the repo on github.
Besides of the content, I want to say thank you to my English teacher Marina Sarg, who helped me on this series of blog a lot. Without her, there won’t be this series of blogs. Marina, thank you very much.
I have been working on Node.js related projects for quite a while, and have built apps with node both for the clients or personal projects, such as LiveHall, CiMonitor, etc. I have promised some one to share my experience on node. Today I’ll begin to work on this. This will be the first blog of the series.
In this blog, I would like to talk about the configuration in node, which is common problem we need to solve in apps.
Problems related to configuration aren’t new, and there have been a dozens of mature solutions, but for Node.js apps, there is still something worth to be discussed.
Perhaps configuration could be treated as a kind of special data. Usually developers prefer to use data language to describe their configurations. Here are some examples:
.net and Java developer usually uses Xml to describe their configuration
Ruby developer prefers Yaml as the configuration language
Data languages are convenient, because developers can easily build DSL on it, then they describe the configuration with the DSL. But is the data language the best option available? Is it really suitable to be used in all scnearios?
Before we answer the questions, I would like to say something about the problem we’re facing. There is one common requirement to all kinds of configuration solutions, which is default values and overriding.
For example, as a Web app default, we use port 80; but in development environment, we prefer to use a port number over 1024, 3000 is a popular choice. That means we need to provide 80 as the default value of the port, but we wish to override the value with 3000 in the development environment.
For the languages I mentioned above, except for Yaml, Xml and Json, doesn’t provide native support of inheritance and overriding. It means we need to implement the mechanism by our own. Take Json as example, we might write the configuration in this way:
The previous Json snippet is a typical example of web app configuration; it has a default section to provide the default values for all environments. Three sections for specific environments. To apply it corecctly to our app, we need to load and parse the Json file to get all data first, then load the values of the default section, then override the value with the values from specific environment. In addition, we might wish to have the validation that yields error when the provided environment doesn’t exist.
This solution looks simple and seems to work, but when you try to apply this approach to your app in real life, you need to watch out some pitfalls.
Issue 1: Confidential Values
In the real world, values in configuration sometimes could be sensitive and need to be kept confidential. It could contain the credential to access your database, Or it could contain the key to decrypt the cookies. It may also contain private certificate that identifies and authenticates the app to other services. In these scenarios, you need to protect your configuration in order to avoid big trouble!
To solve the issue, you might think about adding new feature that enable you to to encrypt confidential values or to load it from a different safe source. To achieve it, you might need to add another layer of DSL which add more complexities to your app and make your code harder to debug or to maintain.
Issue 2: Dynamic Data
A solution to first issue, one could store the environment related but sensitive data in the environment variables. The solution is simple and works perfectly, so I highly recommend it. However, to do this means you need the capability to load the value not only from Json directly but also from the environment variables.
Sometimes, such as deploying your app to Heroku/Nojitsu, might give rise that make the case trickier. After deployed the app to Heroku/Nojitsu, the default values are provided in Json directly, and some of which need to be overrode with the values from environment variables or you need to do it vice versa. These tricky requirements might blow your mind and your code away easily. It causes complicated DSL design and hundreds lines of implementation, but just to load your configuration properly. Obviously it is not a good idea.
Issue 3: Complicated Inheritance Relationship
Scared about above cases? No, then how about complicated inheritance relationship between environments?
In some big and complicated web apps, there might be more than 3 basic environments, such as:
Development: for developers to develop the app locally
Test: for developers to run unit or function test locally, such as mocha tests
Regression: for developers or QAs to run regression tests, such as cucumber tests
Integration: for QAs or Ops to test the integration with other apps
Staging: for ops and QAs to test the app in production like environment before it really goes live
Production: the environment serves your real users
When try to write configurations for these environments, one might find there are only a few differences between environments. To make life easier, to avoid the redundancy, introducing the inheritance between configurations might be a good idea.
As the consequence, the whole configuration becomes environments with complex inheritance relationship. And to support this kind of configuration inheritance, a more complex DSL and hundreds lines of codes are needed.
My assumption above seems to be a little too complex. From some people, it might be the “WORST CASE SCENERIO” and hard to come by. But according to my experience, it is very common when building real web app with node. So if to solve it isn’t too hard, it could be better to consider it seriously and solve it gracefully.
Ruby developer might think they’re lucky because Yaml supports inheritance natively. But confidential data and dynamic data still troubles.
After learnt a number of painful lessons, I figured out a simple but working solution: Configuration as Code – describe the configuration with the same language that the business logic is described!
Configuration as code isn’t a new concept, but it is extremely handy when you use it in node applications! Let me explain why and how it works:
To protect the confidential configuration values, one should store them with environment variables, which are only accessible in the specific server.
Then one can load these values from the environment variables as dynamically values.
To the configuration inheritance, OO languages have already provided very handy inheritance mechanism. Why do we need to invent one? Why not just use it? To the value overriding, OO programming tells us that it is called polymorphism. The only difference here from the typical scenario is that we override the values instead of the behaviors. But it isn’t an issue, because the value could be the result of the behavior, right?
Now I assume that everyone got a pretty good idea of what I am saying. If that is the case, then the below code should be able to be understood quite clearly, which is a standard Node.js file written in coffee script:
See, with the approach, one can describe the configuration easily and clearly in a few lines of code, but with built-in loading dynamical values capability and configuration inheritance and overriding capability.
In fact, with my suggestions, it might work better than expected! Here are the additional free benefits:
Only one configuration is needed when the app deployed to the cloud. Because all the host specific configurations are usually provided via the environment variables in Paas.
Have some simple and straightforward logic in the configuration, which could be very useful, especially if there is some naming convention in the configuration. But complicated or tricky logic should be strictly avoided, because it is hurts the readability and maintainability.
Easy to write tests for configurations, to ensure the values are properly set. It could be very handy when there are complicated inheritance relationships between configurations, or have some simple logic in your configuration.
Avoid to instantiate and execute the code that isn’t related to the current environment, which could be helpful to avoid overhead to instantiate unused expensive resources or to avoid errors caused because of incompatibility between environments.
Get runtime error when the configuration for the environment doesn’t exist.
Besides of the content, I want to say thank you to my English teacher Marina Sarg, who helped me on this series of blog a lot. Without her, there won’t be this series of blogs. Marina, thank you very much.
I was working on the YouTube video playback feature for LiveHall last night, and have it works successfully on my local devbox, which is running Mac OS X. Then I deployed the code to Heroku, without any regression.
This is a very wield issue, then I try the same data on my local dev box once again, and it works perfect! But it does yield error on the production! Then I tried to use heroku toolbelt to run ls command on the production, and I found the there are 4 coffee scripts there.
Then I tried to enforce heroku to redeploy the app by using git push --force, but the issue remains!
Then I even tried to dive into the dependency pacakges connect-assets and snockets, but still found nothing useful.
Same code, same data, but different result! Very odd issue!
After half an hour fighting against the code, I suddenly noticed I the file name is RevealJSPresenter.coffee, whose “S” is capital S! And I reference the file with name #= require ./presenter/RevealJsPresenter, whose ’s’ is a lowercase ’s’!
And snockets depends on the OS feature to locate the file. On my local dev environment, I’m using Mac. Although OS X allow user to explicitly format the HFS+ into file name case sensitive mode, but it is case insensitive by default. So snockets can locate the file even the case is wrong.
But once I have deployed to heroku, which, I assume, runs Linux, whose file system is absolutely filename case sensitive. So the snockets won’t be able to locate the file due to the case issue.
To resolve the bug, I renamed my file in RubyMine, then try to commit in terminal.
But when I commit, I met another very interesting issue, that git says there is no file changed, so it refused to commit.
It is still the same issue, due to FS is case insensitive, git cannot detect the renaming.
This problem is more common when coding on Windows, but CI or production runs on Linux. And the very common solution is to pull the code in case sensitive environment, then rename the file and commit it.
But I found another more easier way to do it:
Using git mv in terminal to rename the file, which will enforce git to track the file renaming action.
Most of Git GUIs are able to track file name case changing, so you can try to commit the change with the tool, such as RubyMine or SourceTree.