Fork me on GitHub

Measure Twice, Byte Once Making Stuff

5Sep/130

Pandoc and MathJax Rendering Issue

Pandoc is a universal document converter. I'm finding it handy for taking technical notes in a combination of Markdown(for typing speed) + LaTeX(for mathematical notation).

Pandoc supports a few different LaTeX renderers, and it seems like MathJax is one of the more common renderers to use. By default, if you're rendering to HTML, specifying the --mathjax argument on the command-line causes LaTeX to be rendered with MathJax-compatible tags and also causes the various MathJax js + support files to be downloaded from a CDN on page load.

In some scenarios you may want to download the various MathJax files from a different location. The --mathjax argument supports an optional URL(which can be a local path) for loading MathJax from a non-standard location. However, one thing that's not obvious from the Pandoc docs is that MathJax actually requires query-string arguments in order to function properly. If you are having issues where LaTeX stops rendering once you specify an alternate URL, this is something to investigate.

In my case, I had to change how I was calling Pandoc from:

pandoc -s --mathjax="file:///home/bla/MathJax.js" Chapter3.md -o Chapter3.html

to:

pandoc -s --mathjax="file:///home/bla/MathJax.js?config=TeX-AMS-MML_HTMLorMML" Chapter3.md -o Chapter3.html

I copied the query-string arguments from the URL that Pandoc uses when no alternate URL is specified. For more information about the MathJax side of things, here's a link to the relevant MathJax ticket. The MathJax docs also provide some more configuration-related options.

Filed under: Uncategorized No Comments
13Aug/130

Upstart/BASH, word unexpected, and line endings

I ran into an issue with some Upstart scripts that would not work on a new server. Trying to start the service resulted in the message "start: Job failed to start". The first step to solving the problem was to look in the Upstart logs under /var/log/upstart. The relevant log contained the message: /proc/self/fd/9: 3: /proc/self/fd/9: Syntax error: word unexpected (expecting "do").

There are other Upstart-related posts about this issue but most of them are Java-related. In our case, the cause of the problem turned out to be that our scripts contained DOS line endings. Although the script was developed on Linux, it was checked into source control and then pushed to the new server from a Windows system. Since our source control system(Git) is setup to convert line endings to the client's native format on checkout, we ended up with a script with DOS line endings.

There are many line ending conversion solutions. I prefer dos2unix, which is available as part of the dos2unix package on Ubuntu.

Filed under: Uncategorized No Comments
31Dec/120

Redis Lua Scripting

I've recently had the opportunity to start using Redis at work. Lua scripting support was added to Redis relatively recently and it allows some pretty slick functionality. Lua scripts execute have performance comparable to built-in Redis commands.

I haven't found too many public Redis Lua example scripts, so I thought I would post a few that I've found useful + some demonstration BASH scripts on Github.

Sunil Arora has a good Redis Lua overview.

Filed under: Uncategorized No Comments
24Dec/121

Amazon S3 URL Signature Generator

Sometimes, it's necessary to share files with individuals via Amazon S3. S3 allows downloads of "private" files by any user as long as the file requests are properly signed. Signing is done with the AWS Secret Key of the user hosting the files.

There are many libraries available to generate signed urls for this use case, but these are a little cumbersome to use for one-off url generation. Therefore, I've put together a Javascript-based tool for url generation, usable here and on Github. These signed urls also have expiration times, making it easier to provide limited access to files even if the urls are shared.

This tool draws from some of the source in the S3 Signature Tester by Chris@AWS at:

Tagged as: , , 1 Comment
4Feb/120

Dan’s House of Pain(t) – Part 2

Part One

Since I was now ready to paint, I started preparing my garage. I used a couple of 10'x20' plastic drop cloths to create a "paint booth". I also purchased a small space heater to help keep the garage warm. Winter is not the ideal time to be painting since the paint tends to need a temperature between 50 and 70 degrees Fahrenheit in order to dry properly. It's also very humid where I live in the Bay Area. We already had a dehumidifier which I moved to the garage to keep the humidity level down.

On painting day(a Sunday), I woke up early, removed the bumper cover, and moved it to the paint booth. I cleaned the entire surface with wax and grease remover. I then applied a few coats of adhesion promoter. This is a clear primer that is designed to help paint stick to different types of surfaces(including plastic). After letting the adhesion promoter dry for about 20 minutes, I started applying the primer. As I started priming, I noticed that the lighting in my garage isn't quite adequate for painting. In addition, I ran out of primer before finishing the second coat. The bumper cover had pretty good coverage, but there were still a few spots that definitely needed more primer. As it turns out, rubberized primer is not carried by any auto parts stores in my area, so I was out of luck trying to finish painting on the same day.

I recommend City Paints(in SF, near 7th and Folsom) if you live in the SF Bay Area and need auto paint. They carry a large variety of auto paint and auto care supplies. Prices were better than most of the online stores that I found. In addition, they will also fill aerosol cans with paint for around $11/can.

The following weekend, I cleaned the bumper cover again and started over with the painting. It turns out that wax & grease remover will eat through rubberized primer(and perhaps also regular primer), so be careful if you need to clean a primed surface. I applied three coats of primer and then wet-sanded with 400 grit paper.

First coat of primer

First coat of primer

Setup for wet sanding outside

Setup for wet sanding outside

I started out with the bumper cover supported by saw horses and facing up. It turns out that it's much easier to paint a vertical surface than a horizontal surface, especially with aerosol paint. I ended up using a rope to suspend the bumper cover from the ceiling for the color and clear coats.

Hanging setup for painting

Hanging setup for painting

I then applied color coats which I let dry overnight. The final color coat should not be sanded, but earlier color coats can be sanded. One of the issues with metallic colors is that the thickness of the color coat affects the final color and appearance of the paint job. If I were redoing the process, I would sand between the second and third color coats since there are a few areas where the paint ended up uneven.

The following morning I applied four clear coats, let the bumper cover dry, and then put it back on the car. The clear coat has to dry for several days before it can be polished. An additional thirty days are required for drying before the surface can be waxed. The bumper cover color matches the rest of the car very well. In the interest of potentially saving money in the future, I also bought a can of Dupli-color paint from my local auto store. Dupli-color paint is around $7/can, whereas the paint I purchased from paintscratch.com was around $20/can. The Dupli-color paint also matches the car reasonably well, but the match is not quite as good as the paintscratch.com paint. Most people would probably not notice the difference in color unless it was pointed out to them.

I was able to complete the job with two cans each of primer, color, and clear coat. I would recommend three cans since two leaves a pretty small margin for error and repainting.

Since I was saving a fair amount of money by repairing the bumper cover myself, I was able to buy a nice auto polisher and supplies from Griot's Garage. A week after painting, I first wet-sanded with 1000 grit paper. I then used some aggressive polishing compound with the polisher to polish the bumper cover. This leaves a shiny surface, although it doesn't quite match the rest of the car in terms of shininess. I'm hoping that the eventual sealant and wax will take care of this issue.

Part One

Filed under: Car, DIY No Comments
4Feb/120

Dan’s House of Pain(t) – Part 1

Part Two

It all started with a bang. My car was involved in a minor accident and sustained some damage to the bumper. The initial repair quote was a minimum of $800($200 for a new bumper and the remainder for labor). Rather than paying someone else to have fun fixing my car, I decided to do the work myself. In addition, since I wasn't driving the car at the time of the accident, the driver gets to pay for my new tools, supplies, and fun…perfect!

Damage

Damage


The first step was to assess the extent of the damage. The grille was partially broken during the accident and the Toyota emblem(expensive!) was lost. Although most people refer to the front of their cars as the "bumper", on newer cars this is technically just the bumper cover. The bumper cover is a thin piece of plastic that goes over a tubular steel structure and some supporting foam.

Damage to foam and bent supports

Damage to foam and bent supports

My bumper cover paint had been scraped down to bare plastic in a few places. The impact also forced the license plate into the bumper cover causing some deformation. It's also noticeable in the pictures that the bumper had been pushed in and no longer lined up with the headlights. I removed and cleaned the bumper cover and took a look at the supporting structures. On my car(2003 Camry), there is a horizontal metal support that runs between the headlights and under the top of the bumper cover. This support was clearly bent out of shape. There is an additional vertical support that runs up the center of radiator. This support also appeared to be bent in some way, but I was unable to determine how much. I attempted to straighten the horizontal support, but without a reference I was unable to make much progress. The first step then was to order a new horizontal support and grille. I was able to purchase the support and grille from autopartswarehouse.com for $58(including shipping).

New support and old support

New support and old support


Otherwise, the car appeared fine. Fortunately, the headlights were not broken since these tend to be expensive to replace.

I ordered a couple of books on auto restoration and painting and started doing some research. I wanted to determine if it would be possible to paint the bumper cover at home. Professionals use an air compressor and spray gun for paint jobs, but the average home air compressor can't do the job properly. However, it is possible to get matched auto paint in aerosol spray cans(paintscratch.com), which makes painting more feasible. As a result, I decided that I would repair and paint the bumper cover myself. As with any paint job, the secret to success(and the bulk of the effort) is in the preparation

Since the bumper cover is plastic, hammering out the dents wasn't an option. I would have to use a specially designed bumper filler(two-part epoxy) by 3M/Bondo to create a smooth surface. Many traditional auto repair fillers and primers cannot be used with bumper covers because these products are firm and bumper covers are flexible. If normal filler is used, it's possible that the entire repaired area may separate from the bumper cover when it flexes.

The filler only adheres adheres to plastic, so I used a random orbital sander with 100-grit sand paper to strip away the paint. As it turns out, It's slightly liberating the first time you take a sander to your car. Whether the car will look good at the end is something else entirely. I used an automotive wax + grease remover(smells like super rubbing alcohol) to prepare the surface for filling. There were a few areas that needed a fair amount of filler, so I ended up making the repairs over a few weekends and probably spent between 5 and 10 hours on this part of the project. The filler has a working time of a few minutes and is sandable after 20-30 minutes. I used a long, flexible sanding board to ensure that the filler that I was building up would match the overall curve of the bumper.

Eventually, I reached a point where the bumper was starting to look pretty good. There were still small holes in some places in the filler. I switched to using two-part glazing putty. This is initially much more fluid than the filler and can be spread to a much smoother finish. After a few rounds of putty + sanding, I was done with most of the repair work. I then wet-sanded with 220 grit paper to take out any leftover imperfections in the filled areas. I also wet-sanded the rest of the bumper cover, since new paint adheres better to a surface that has been scuffed.

Initial round of filling and sanding

Initial round of filling and sanding

Done filling and sanding

Done filling and sanding

Part Two

Filed under: Car, DIY No Comments
19Jan/120

Android NativeActivity JNI and ClassNotFoundException

We're working on some code related to the Android NDK and using JNI to call Java methods from C/C++. This site has some great code + explanation on how to access your custom Java classes. If you still get ClassNotFoundExceptions after implementing that code, and you've tried everything you can think of, double-check your AndroidManifest.xml for the application attribute android:hasCode="false".

Some of the NDK samples include this setting, which will cause your Java code to be left out of the final APK.

14Jan/124

Fixing ProGuard Warning “can’t write resource [META-INF/MANIFEST.MF]“

We're using ProGuard in our Android build and were getting a number of warnings regarding duplicate entries in the output jar. When ProGuard encounters resources in an input jar, it will by default copy the resources into the output jar. This is fine as long as you don't have multiple jars that contain resources with the same name. All jars(that I'm aware of) have manifest files, so attempting to use ProGuard with multiple input jars will cause warnings that look like: "Warning: can't write resource [META-INF/MANIFEST.MF] (Duplicate zip entry [yyy.jar:META-INF/MANIFEST.MF])".

The warnings are harmless, but a clean build is a happy build, so we might as well try and get rid of them. The ProGuard Manual actually describes what needs to be done, but it only provides a solution for the case where input jars are specified individually:

When using ProGuard in the Android build process, the default behavior is to pass in all input jars in a giant delimited(semi-colon on Windows) blob. I have done some experimentation, but was unable to determine how an input filter can be used when multiple jars(having absolute paths) are passed in this way. Fortunately, it's not too difficult to change the build process to pass in a list of input jars individually and with individual filters.

This is the relevant section of the standard Android build.xml(starting around line 700):

The directives above create a new object that combines the location of the compiled jar along with other jars that were used during compilation. These locations are then joined into a string by the pathconvert task, which also does some transformation to ensure that paths containing spaces are quoted properly. Finally, the string from pathconvert(stored in project.jars) is passed into proguard via the statement: "-injars ${project.jars}"

Update(2013/02/11): paulpv has provided an updated config since the Android build system has changed since my original post. I have included his config here:

<?xml version="1.0" encoding="UTF-8"?>
<project name="custom_rules">
    <!--
    Suppress "Warning: can't write resource [META-INF/MANIFEST.MF] (Duplicate zip entry [classes.jar:META-INF/MANIFEST.MF])".
  	Per http://www.dancartoon.com/2012/01/14/fixing-proguard-warning-cant-write-resource-meta-infmanifest-mf/
  	Target "-obfuscate" copied from ${sdk.dir}/tools/ant/build.xml v21.
  	-->
    <target name="-obfuscate">
        <if condition="${proguard.enabled}">
            <then>
                <property name="obfuscate.absolute.dir" location="${out.absolute.dir}/proguard" />
                <property name="preobfuscate.jar.file" value="${obfuscate.absolute.dir}/original.jar" />
                <property name="obfuscated.jar.file" value="${obfuscate.absolute.dir}/obfuscated.jar" />
                <!-- input for dex will be proguard's output -->
                <property name="out.dex.input.absolute.dir" value="${obfuscated.jar.file}" />
  
                <!-- Add Proguard Tasks -->
                <property name="proguard.jar" location="${android.tools.dir}/proguard/lib/proguard.jar" />
                <taskdef name="proguard" classname="proguard.ant.ProGuardTask" classpath="${proguard.jar}" />
  
                <!-- Set the android classpath Path object into a single property. It'll be
                     all the jar files separated by a platform path-separator.
                     Each path must be quoted if it contains spaces.
                -->
                <pathconvert property="project.target.classpath.value" refid="project.target.class.path">
                    <firstmatchmapper>
                        <regexpmapper from='^([^ ]*)( .*)$$' to='"\1\2"'/>
                        <identitymapper/>
                    </firstmatchmapper>
                </pathconvert>
  
                <!-- Build a path object with all the jar files that must be obfuscated.
                     This include the project compiled source code and any 3rd party jar
                     files. -->
                <path id="project.all.classes.path">
                    <pathelement location="${preobfuscate.jar.file}" />
                    <path refid="project.all.jars.path" />
                </path>
                <!-- Set the project jar files Path object into a single property. It'll be
                     all the jar files separated by a platform path-separator.
                     Each path must be quoted if it contains spaces.
                -->
  			<!--
  			Old:
  			<pathconvert property="project.all.classes.value" refid="project.all.classes.path">
  			New:
  			-->
                <pathconvert property="project.all.classes.value" refid="project.all.jars.path" pathsep=" ">
                    <firstmatchmapper>
  					<!--
  					Old:
                        <regexpmapper from='^([^ ]*)( .*)$$' to='"\1\2"'/>
                        <identitymapper/>
  					New:
  					-->
  					<regexpmapper from='^([^ ]*)( .*)$$' to='-injars "\1\2"(!META-INF/MANIFEST.MF)'/>
  					<regexpmapper from='(.*)' to='-injars \1(!META-INF/MANIFEST.MF)'/>						
                    </firstmatchmapper>
                </pathconvert>
  
                <!-- Turn the path property ${proguard.config} from an A:B:C property
                     into a series of includes: -include A -include B -include C
                     suitable for processing by the ProGuard task. Note - this does
                     not include the leading '-include "' or the closing '"'; those
                     are added under the <proguard> call below.
                -->
                <path id="proguard.configpath">
                    <pathelement path="${proguard.config}"/>
                </path>
                <pathconvert pathsep='" -include "' property="proguard.configcmd" refid="proguard.configpath"/>
  			
  			<echo>proguard: $${project.all.classes.value}=${project.all.classes.value}</echo>				
  
                <mkdir   dir="${obfuscate.absolute.dir}" />
                <delete file="${preobfuscate.jar.file}"/>
                <delete file="${obfuscated.jar.file}"/>
                <jar basedir="${out.classes.absolute.dir}"
                    destfile="${preobfuscate.jar.file}" />
                <proguard>
                    -include      "${proguard.configcmd}"
                    -include      "${out.absolute.dir}/proguard.txt"
  				-injars       ${preobfuscate.jar.file}
                    ${project.all.classes.value}
                    -outjars      "${obfuscated.jar.file}"
                    -libraryjars  ${project.target.classpath.value}
                    -dump         "${obfuscate.absolute.dir}/dump.txt"
                    -printseeds   "${obfuscate.absolute.dir}/seeds.txt"
                    -printusage   "${obfuscate.absolute.dir}/usage.txt"
                    -printmapping "${obfuscate.absolute.dir}/mapping.txt"
                </proguard>
            </then>
        </if>
    </target>
</project>

The goal of my changes is to preserve the manifest files from the original jar("preobfuscate.jar.file") and ignore(filter) the manifest files from the remaining input jar files. Since I haven't figured out how to filter input jars when passed in as a blob, I instead use the pathconvert task to transform each input jar location into a separate "-injars" statement with its own manifest filter. The property containing these locations is then passed into ProGuard directly via: "${project.jars}"

The solution isn't as clean as I would have liked, but it does fix the warnings.

17Jul/110

Implicit vs Explicit Returns in Ruby

Ruby supports implicit returns, i.e., the value of the last expression in a block/statement is what will be returned.  One might assume that  implicit returns and explicit returns(e.g. "return 123") are functionally equivalent.  Although this is often the case, there is a gotcha scenario when using an explicit return within a block.  In the following Gist, an explicit return is used within a map block.  The result is that explicitReturn() will end up returning a single upper-case string("PANGOLIN"), rather than an array of upper-case strings.

On a semi-related note, Martin Hsu provided my Pangolin inspiration: http://www.martinhsu.com/pangolin-rider-p-50.html

Filed under: Coding, Gotchas, Ruby No Comments
2Jan/111

Fractal Rendering

I had some vacation time recently and wanted to explore HTML5/Canvas and fractals, so I wrote a basic JavaScript-based Mandelbrot renderer. The code is available in my GitHub repository, and the demo page is here. It takes about 10-15 seconds to render a single scene. This would be a good application for parallelization, but there's generally only a single thread available to JavaScript within the browser. Perhaps there's an opportunity to add support for HTML5 worker threads here.