Wednesday, April 4, 2018

Organ Pedalboard MIDI Project

I like to play the organ, and am the designated organist for my church. With just a digital piano, I can practice the hands (manuals) at home, but not the feet (pedals). Having to go to the church building is enough of a hurdle that i just didn't end up practicing as much as i wanted. So i decided that i would do as many others have done and build my own pedalboard setup at home.





Of course i didn't want to do this using analog tube, pre-amps and other old hardware, but rather using a the midi protocol.
Salvaged analog tubes and other relays used to power the organ i purchased.


Pre-amps

Essentially this means that because my pedalboard is just sending data and not sound, I can use any MIDI synthesizer which knows how to read this data and produce sound. In other words, with just the press of a button I can produce organ sounds from any organ someone has recorded and made available like from this site.



 I could even make it sound like a bass guitar or grand piano, much like a electric keyboard. That's the power of MIDI!



From start to finish this project took me about 5 months. However, given what I know now, I could probably do it all again in 1 to 2 weeks.

Step 1: Obtain a pedalboard.


There are some examples of people actually building their own pedalboard from scratch, but this was way too much work for me and i also don't have the equipment. Instead i found a very used organ at a local thrift store for $75. The trick is to find one that has the full pedalboard and not a half. Of course i didn't need the manuals or the tubes or other components, so i salvaged just the pedalboard a few other wiring pieces i though might come in handy. Sadly i didn't get a picture of my organ before i tore it down for parts. But it was pretty close to this.

Mine also didn't come with a bench :(

These are basically the only parts i kept.

My Z-stand will no longer work to hold my piano with the pedals, so i had to build this new stand. I also constructed a covering for the bottom portion of the pedals so that i could sit my piano bench on top of it. 



Step 2: Create a midi controller

A midi controller is the piece of hardware that takes the inputs from a pedalboard and converts this mechanical action into midi signals. Since i had very little idea where to start I copied this project hosted on Instructables.


Since this blog is also about posting my experience and not just the final product, step 2 also contains the details of my first attempt (version 1) and how i arrived to the final product (version 2). If you only wish to see the steps leading to the final product, click here.



Version 1

It listed all the steps and equipment needed which i purchased and waited for. To build this project i purchased the following materials totalling somewhere between 20 and 30 dollars

  • Perf board (later replaced with breadboard)
  • pull down resistors
  • jumper cables
  • 2x 16 MHz crystals
  • 4x capacitors
  • 2x ATMega328
  • 2x soldering seats (which i never used)
  • and A midi to usb cable

In addition, during my time trying to get this setup to work, i also purchased the following for debugging and flashing purposed
  • a USBasp (flashing the ATmegas with a bootloader and sketch files) $8
  • a Salae based logic analyzer (debugging signals) $6
The majority of time spent on the project was trying to get this setup to work. I spent a lot of time going over the wiring diagram and verifying that my breadboard was wired correctly.

Some initial breadboarding and getting my feet wet with this Arduino business
"Upgrade" to a perf board. My soldering iron could not handle this type of precision.


At the top of this picture is what i refer to as the serial harness. An excellent piece that will come in handy later.


 In addition because this was my first encounter with Arduino, there was also a learning curve about how to flash the ATMegas. Supposedly i could have flashed them with just another Arduino, which i already had, but when that didn't work i decided to purchase the USBasp. After i also couldn't get that to work i reverted to the Arduino/breadboard approach and finally, after discovering that my breadboard was either broken or designed so as not to have the ground and power lines connect all the way down the board, I flashed the ATmegas with the bootloader. Hoping I was on the final step, I deployed the sketch file and expected to get some signal when completing the circuits. but alas, it was as if the loop wasn't running. Hence i bought the logic analyzer to debug the traffic being sent across the wires. To my surprise i did see traffic but it was not correctly encoded in midi protocol. A coworker suggested that it could be the clock speed or a number of other things that i was about to dive into. He then also suggested that perhaps i buy a different Arduino that already has a clock and pull down resistors built in. This was the break i needed. I instantly bought an Arduino Mega ($14). 


Arduino Mega with enough pins for each pedal

The project mentioned before uses two Arduinos (ATMegas) because there aren't enough pins on a single Arduino to accommodate all 25 pedals. Although i had two regular Arduino Unos, it was unsightly to try and mesh them both together with the pull down resistors, capacitors, crystals, and bread board. The arduino mega didn't have this problem as is has 50 pins. Having dug into the MIDI protocol and sketch file by this point, I was comfortable altering the two sketches (one for each of the ATMegas) into one sketch that included all 25 pedals.

Using an Arduino instead of a chip also greatly decreased time to deploy any updates to my sketch since i didn't have to change the wiring on the breadboard.

Version 2

Since i was now using an actual Arduino and not an ATMega, i no longer needed any physical pull down resistors (since software pull-up resistors can be enabled in code), the crystals or capacitors. In short I removed everything on the bread board and simply plugged in the wires from the pedals directly into the Arduino board ports 22-47. Connecting the ground pin into any of these pins 22-47 will produce a MIDI signal.



But we only want signal when the pedal is pressed, so we just need to have ground connected to each of the pedals. For me, i reused the salvaged organs wires and setup.
The wire at the bottom gets connected to ground. Later i moved it closer to the rest of the wires just for a cleaner setup.




In the last picture found in the Version 1 section you can see that all the wires are fed into a serial harness. Below you can see the wires from the pedals to the other end of the harness. This is done so that i can disconnect the Arduino from the wooden pedals easily without rewiring. I highly recommend this approach.
Here are the wires going to the pedals. Also terminated at the serial harness. So that when the two are connected the signal transfers through, but disconnecting is easy if i need to move the Arduino.

Rather than trying to line up and keep track of each pedal and which pin it was physically connected to, i decided to just chang the assignments in code. 
The commented numbers on the side were the assignments i made after my first attempt through the process described below. Most were right, but some i had to change several times.



So if you copy this code, your keys will all be mapped incorrectly -- A tedious and time consuming, but simple problem to fix. I just assigned all the notes to the first pin number and when pressing a pedal actually fired a signal i knew that pedal (note) should be assigned to that pin. Then i changed all the notes in code to the next pin number up. Repeat this process until you've mapped all pedals to the right pin number.

In part of the code you will also see that pin 52 is setup for output. This is for the LED that lights up whenever a note is being played.



Lastly, now that the Arduino is seeing the pedal pushes and creating a midi signal, we need to actually transmit the MIDI data.


This cord worked for me by connecting the Arduino Tx pin to the middle pin on the midi cable and ground to ground. The usb then goes into a midi synthesizer and voila - pedal presses make sound.


Step 3: Create a midi synthesizer

Although i could connect my Macbook pro to the usb midi cable every time i want to play organ, it would be better to have a dedicated midi synth that is always connected. I didn't need all the power of something like Zynthian. So using a Raspberry Pi, here is an open source project call SampleBox for a MIDI synth that you can load your own sample sets onto.  Here is a link to a sample set that i found and modified for the C0 to C2 range.

Here you can see the Raspberry Pi receiving data from the USB MIDI cable. The sound then gets output to a small amp.


Next steps:

Some additional things i'd like to do next are

  • Add an expression pedal
    • This should be easy since there are plenty of pins left and i still have the expression pedal/potentiometer
  • Transmit midi data from Arduino pin to raspberry pi pin instead of through usb midi cable
    • Samplebox apparently has this feature, so it's just a matter of enabling it somehow.
  • Add a sound card to the Rpi synth for better sound quality
    • On its way. Hopefully this will drastically improve the sound quality.
  • Add new sample sets which can be easily cycled through using the Program change MIDI command.
    • I added a progChange method to my Arduino sketch, but haven't decided the best way to trigger it. I also haven't curated any additional sample sets, so who knows if i'll get around to this. After all, this is just a practice setup for me so i don't really need other sounds.


Thursday, March 8, 2018

Samsung Smartthings: Home automation system upgrade

Well it's been more than a solid year since my last post and in that time I have become much more invested in home automation. With a slew of new products and buzz this market is exploding with all kinds of setups. My biggest frustration with Home Assistant was three-fold:

  • Adding devices - This would usually require looking up the specific config type and then diving into terminal and updating my yaml config files. For z-wave you would also have to go through the inclusion process using the semi-awful ozcwp. Furthermore I found myself restarting Home Assistant about a thousand times during that process. 
  • Automation - Although the rules, triggers, events and everything else that go into the automation portion are nearly endless, so is the documentation (not to mention keeping up with which documentation was current). For me it was all very confusing and even harder to implement because the debugging and other tools are just not great.
  • Reliability/performance of state - My most common use case was and still is turning off the lights for a movie or bed. Telling my Google Home to turn off the light worked great because i knew the light was on as i was in the room. However if i pulled out the app i would notice that 9 times out of 10 it would be wrong. The icon would show that the light was on when in fact it was not. Being at home, this really isn't a problem because as i said, the events still worked and executed properly. But if i wasn't home, i couldn't really know. My usual workaround was to press it twice. Then i could safely assume that the light was in fact turned on (or remained on). Then repress the icon and turn it off. Increasing the polling frequency (despite warnings about flooding the network) didn't help.
After a while i started to notice that even though i was getting new devices, the dreadful task of adding them to the network or setting up automations brought out the procrastinator in me. After all, home automation is about making your life easier, right?

When the Samsung Smart things hub went on sale from $99 to $49 i pulled the trigger and decided to make the switch. I was ready for a GUI for setting up new devices and automations. Even the worst GUI would still be easier than reading through documentation, or searching for examples of others' configuration, copying to my own via terminal, then testing by restarting the whole system.

Luckily the Smart things app is not the worst GUI. In fact it's pretty good with just the built-in pieces. Devices are added automatically once in pairing mode with all the device properties they need. For example, I don't need to specify that GE_Z_WAVE_DIMMER is a light switch and can do dimming. It just works. It's compatible with most z-wave and zigbee products which meant that only one of my off-brand z-wave devices can no longer be used. 

Automation is also a snap because it's GUI driven. Samsung publishes Smart Apps that are available for most common use cases and the community can contribute for less common or more advanced use cases. But the best part is that i still have full control for really custom use cases. For example, my garage door opener is a raspberry pi running home-built software. I did come pretty close to finding a solution that would work in the community device type "URI switch" but since it only had properties of a switch and not a garage i couldn't do some additional things. So i went ahead and built my own device type handler. Since I was able to classify it as a garage door it had all the same functionality as a $600 plug and play garage door opener such as the Chamberlain MyQ (which is actually not supported by smart things currently)


Some things you lose by switching:
  • Compatibility. For example, I can no longer use the Nest through smart things because Nest does not allow apps to integrate unless they've been approved. Open source projects like Home assistant get around this by having you create your own Nest developer account and self-publishing an app. 
  • You sort of lose compatibility with hardware. Whereas anything under the sun can be configured to work with Home Assistant, Smart things has recommended cameras, light switches, etc. I say "sort of" because if you really want to, SmartThings does allow you to really add anything you want like I mentioned with my Garage Door opener. And in reality this isn't so much of a loss because things just work and you don't have to do the configuration yourself. 
On a final note, I'm excited to see where Samsung is taking the new SmartThings app as they've just announced that it will be merged with Samsung Connect. I have seen some users complain that not all Samsung products are supported with SmartThings. Indeed, the controls for my own 2016 Samsung Smart TV don't really work after adding it to the SmartThings app. But it did work with Samsung Connect the one time i tried.


Have you tried other systems? What's been your favorite?

Thursday, November 3, 2016

Home Assistant

Wanting to expand my home automation setup, it was time to find a unifying infrastructure to tie it all together. The open source community project Home Assistant is a great tool for just that. However it can be a lot of frustrating work, as things can easily go wrong and the amount of forum posts is somewhat pitiful. This is my attempt to document some of my experience in order to save some time during my next installation.

My first piece of advice is to use the raspberry pi all in one installer. Only after spending about 10 hours trying to get it setup for my regular linux server did i finally give in and switch. About 2 hours  later (most of which was simply waiting for the installer) I had working installation on my rapsberry pi 2 with all the right permissions and virtual environments. It even had open z-wave control panel ready to start, which I would definitely be using.

Open z-wave control panel (ozwcp), was by far the most difficult part to get running correctly and then use. The good news however is that once you know how it works and what it's for, it's quite easy to reuse. Indeed i just added a new light switch to home assistant using ozwcp in under 5 minutes.
The purpose of ozwcp is simply to provide a gui for writing the z wave configuration xml file that home assistant uses later. These are the steps i have found to be most simple and effective.

Step 0: I find it's easiest to "include" my new z wave module into my z wave setup by doing this outside of ozwcp. Using the Aeon Labs Aeotec Z-Wave Z-Stick, Gen5 (ZW090) simply unplug the transmitter from your box and get in range of the new device. Press once to enable pairing mode, then pair with new device. Hopefully your device comes with instructions on how to include it in a zwave network. If successful, plug your transmitter back into the hass box and continue to step 1. If not, your new device probably doesn't have power/see other troubleshooting advice.

Step 1: Stop Home Assistant, either through the web app rest calls, or by killing the service on the box it's running on. If you can no longer hit the web page and instead see a 503, this should be enough.

Step 2: Start the ozwcp executable. In order to do this you'll need to ssh into your box running hass and cd to the directory where the ozwcp executable file exists. For me that is

cd /srv/hass/src/open-zwave-control-panel/

then run the executable on some port

./ozwcp -p 8888


Step 3: Point your browser to ozwcp 192.168.1.{hassBox}:8888

Step 4: Initialize your z wave transmitter device by looking up it's tty listing or by entering the id. There are plenty of instructions on how to do this, so i'll just skip to showing you what mine looks like.

/dev/serial/by-id/usb-0658_0200-if00

Theoretically this reference by id should never change and thus should always work. But sometimes i've found that it does not. In those instances, i tried looking up the tty listing (ls /dev/ttyACM*) and using that entry which then works. Remember not to check the usb button even though clearly mine says usb in the id.

Step 5: This is the confusing part because no one ever explains that after initializing you should see your z wave transmitter and any other device that's already included in the network. Granted, it's only confusing when it doesn't work, cause you are given no helpful directions as to what you should do next.

 
If nothing appears, something has already gone wrong and there is no point in playing with the buttons to try and make things appear. Some possible reasons are that your transmitter is not plugged in or needs to be reinserted. Make sure it shows up in the ttyACM* list. You could also try digging through the error logs, but the errors codes are completely cryptic and usually unhelpful.

As I said, when it works, it's pretty easy. All you have to do now is customize your devices. Use the functions drop down to set a name or location. Also note customizing is optional, but you should definitely be seeing your devices before moving on.

Step 6: This step is not optional and very important. Make sure you press save under the Backup Controller menu. This is what actually writes your devices to the z wave xml that home assistant looks up.

Step 7: Close ozwcp and start hass
sudo systemctl start home-assistant.service

Step 8: If discovery is off, add a configuration entry and name it however you want.

sudo nano /home/hass/.homeassistant/configuration.yaml

Once it boots up and you can see the web page, look under the dev-state page to see your device. You'll need to use the Entity name when referencing this device in your hass config. Usually it's something like light.__level_4_0 or switch.vision_zl7432_inwall_switch_dual_relay_switch_3_0

Here's an example configuration entry.

customize:
    switch.vision_zl7432_inwall_switch_dual_relay_switch_3_0_2:
      friendly_name: 'Master Lights'
      icon: mdi:lightbulb



Fortunately there are plenty of examples and the documentation is pretty well covered on what things mean.



That's it!





Friday, February 19, 2016

Are you still opening your garage door with a button?


This project's goal was to be able to automate my garage door so that instead of only being able to use the wall button or a car remote, I can send a signal from anywhere and the door will open or close. 



When I told people my idea to automate my garage door, many people felt it was not a problem that needed to be solved. And they're right. But when home automation is your hobby, eventually you run out of necessary things to automate. It turned out to be a lot of work -- a lot more than pushing the button ever was. But having had it setup for a couple weeks now. It's pretty fun to use and so nice to be able to drive right into the garage without waiting that extra 5 seconds.

Waiting 5 seconds for your door to go up an average of just two times a day, adds up to 1 hour of waiting each year!



Automating my garage door has easily been the most elaborate project I've ever worked on. That's because, the only easy solution is to buy a smart garage door opener which cost about $600+. My solution cost me about $15 because I had most of the equipment already including

  • a raspberry pi
  • wifi adapter
  • sd card
  • pi enclosure
  • power supply (usb micro cord and wall adapter)
The equipment I had to buy was


Starting from scratch with none of these materials might set you back about $80. But my solution was created specifically because I had most of the equipment already. That, and now I finally have a "sweet" use for my Raspberry Pi.



Figuring out the hardware was tricky for me because I don't have much experience in small electronics. Nevertheless, there are basically four main elements to this solution.

  1. Configure the raspberry Pi to act as a REST server and forward signals to the pins
  2. Wire up the pin signal to trigger the opening and closing on the motor. 
  3. Add a sensor to detect the current position of the door. Without this, you cannot build reliable application logic.
  4. Expose the server endpoints and write an app that can trigger them either through some UI buttons or a geofence, or both.



Finished Product

Part 1

Configure the raspberry Pi to be a server

I did some investigation and found a wonderful library called Webiopi that did exactly what I needed. With just a few lines in the terminal you can setup a server just for REST calls or to host an html site. Since all I needed was REST, I left most of the configuration file to its default settings and just specified a python app to load. This framework was perfect for two reasons:

  • To expose the methods for my rest layer,  I simply annotated the methods with @webiopi.macro
  •  It had built-in classes to configure and send signals to the GPIO board.

Here's the python app I load into the service at launch.


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
import webiopi
import datetime
from webiopi.utils import types

GPIO = webiopi.GPIO
LIGHT = 23
SENSOR = 27
 
def setup():
 GPIO.setFunction(LIGHT, GPIO.OUT)
 GPIO.setFunction(SENSOR, GPIO.IN, GPIO.PUD_DOWN)

def loop():
 webiopi.sleep(3)

def destroy():
 GPIO.digitalWrite(LIGHT, GPIO.LOW)

def activate():
 GPIO.digitalWrite(LIGHT, GPIO.HIGH)
 webiopi.sleep(2)
 GPIO.digitalWrite(LIGHT, GPIO.LOW)

def isOpen():
 open = GPIO.digitalRead(SENSOR)
 print("Current state is:")
 if open is True:
  print("OPEN")
 else:
  print("CLOSED")
#returns true if voltage is high
 return open

@webiopi.macro
def getState():
 json = {}
 if isOpen() is False:
  json['closed'] = True
 else:
  json['closed'] = False
 return types.jsonDumps(json)

@webiopi.macro
def openDoor():
 if isOpen() is False:
  print("OPENING")
  #json = getState()
  activate()
 return types.jsonDumps({"action":"Opening"})

@webiopi.macro
def closeDoor():
 if isOpen() is True:
  print("CLOSING")
  #json = getState()
  activate()
 return types.jsonDumps({"action":"Closing"})
Using the @webiopi.macro annotation, I have exposed 3 methods, openDoor(), closeDoor() and getState(). Being able to receive input on the SENSOR variable allows my code to do more than execute activate() as I leave the house and hope that the signal was sent without errors.

By having this state information I can confidently send a closeDoor() command without worrying that I'm actually opening an already closed door. You'll also notice the setup(), loop() and destroy() methods. These are part of the webiopi framework and you can override them with any custom logic. In my case I simply needed to initialize the pins I would be using later for input and output. Later on, I'll talk about why that third argument (GPIO.PUD_DOWN) is necessary.

To finish off part 1, I had to figure out how to call these methods via rest. Here's how you would hit the getState() method.

POST /macros/getState HTTP/1.1
Host: 192.168.1.17:8000
Authorization: Basic {{username:password}}

As you can see, the path for rest endpoints simply starts at /macros/ and you must include your Authorization header with the base64 encoded username and password (default is webiopi:raspberry). If you want to hit this from outside your home network, you'll need a domain or static IP and port forward 8000 to hit your raspberry pi. Also make sure you use POST, instead of GET (even though it should be a GET). Perhaps there is a more appropriate annotation, but I don't care.


Part 2

Signals from pins activate the motor

This part is actually pretty simple once you have the right equipment. In the code above, pin 23 (variable name LIGHT) was activated anytime the door needed to change. If you don't know anything about relays, they are essentially a way to turn a small electrical signal into a stronger one. In this case, the Raspberry Pi has a 3.3v output pin and the garage door opener might need something stronger than that (though I actually never tested it going direct, which is why I mentioned earlier, that you might not need the relay). Here's a diagram labeling how the pins were connected to the relay. Ground and 3.3v were arbitrary/whatever was available.

The GPIO board of the PI with pins connecting to the relay


The relay then connects from the NO and COM directly into the same ports as the wall button wires. This makes the solution especially convenient when it's time to move. Simply remove the wires and nothing has changed. Also, with this configuration, I can still use all the same remotes and buttons that currently exist on the system.


The Green wires come from the relay and fit right alongside the wires from the wall button. No splicing required.


Part 3

Detect current position of door

This part took a while to get right. Not because the solution is complicated, but because I didn't understand it very well. To read the state of the door, I used a magnetic switch. When the magnets line up, the signal can flow. So you create a circuit from the board, through the switch, then back to the board. Connect a 3.3v pin to the COM on the switch, then from NC on the switch back to the input pin that you've registered in your program. From my code you can see I setup pin 27 (variable named SENSOR) as my input during the setup loop. NC stands for Normally Closed, because our door will normally be closed. Though technically you could use NO, if you reversed your application logic.

GPIO Board showing connections to magnetic switch


When I got to this point I started testing the input signal. Using the GPIO dashboard that comes with webiopi, I noticed that the input pin would constantly flip between on and off regardless of whether the magnet was aligned. To solve this problem I had to be educated about pull down resistors.

<Pull Down Resistor Explanation>

The third argument (GPIO.PUD_DOWN) is necessary because the signal that will flow through this small circuit is analog. An analog signal doesn't produce a 0, it produces a current slightly less than the 3.3v. But since the pi is expecting a digital signal of 0, it switches back and forth between on and off. A pull down resistor is an actual piece of hardware that just acts like a buffer for all the low voltages. Until the buffer fills, by receiving a high signal, the resistor will pass on a 0. Luckily the PI has this resistor capability built in and it simply needs to be configured to use it, hence the third argument.

</Pull Down Resistor Explanation>

Once I could safely rely on the signal, I placed the magnets. The rubber track/chain ended up being the easiest place to mount a magnet that wouldn't fall and that would line up with my receiver.

The magnet switch
When aligned, the current can flow. A "high" signal tells us that the state of the door is closed.


The stationary portion of the magnet switch connects to the pi and lines up directly with an ordinary magnet (taped to the rubber track) only when the door is fully closed. I recommend establishing the "lined up" state to a closed door instead of open, as this will greatly reduce your risk of a false positives.

Part 4

The mobile app. Lest what's all this been for?

Finally I was back in my comfort zone. A simple UI with a few buttons (like Open, Close and Get State) was enough to put a smile on my face.


But I also wanted to have it be fully automated. As soon as I would get within a block or so of my house, the garage would just open. For this to happen, I needed to create a geofence.



Geofences work like this on Android:

  • You setup a circular virtual fence around a center coordinate. 
  • You tell it that you want to watch for an entry, exit, or dwell event (or all three). 
  • You attach a PendingIntent to the trigger of any of these events. 
  • You submit this to the Android geofence directory. 
  • The system then takes care of launching your pending intent when any of the events you registered for happen. 
  • During the execution of your pending intent, you can see which type of event was the trigger and from there determine the appropriate logic. For example, on an entry event, I would want to call the open function, and for an exit event, the close function. 


A couple things to note:

  • The GPS sensor must be active in order to realize it's inside a geofence. This has resulted in me opening the maps app as I pull in to the last stretch of my drive home so that it's constantly refreshing. So much for being fully automated. 
  • When you submit your geofence to the Android geofence directory, it will immediately fire off your pending intent if any of the events are active. In other words, if you are inside your geofence while submitting, it will fire off an entry event. This was the reason I stopped registering the geofence on app launch. Because while coding at home, with each restart of the app, my garage door would open. 
  • Lastly, from what I've read, the OS clears the list of registered Geofences with each restart. So instead of registering the same geofence with each app launch, it's inside a menu option which I will have to periodically press whenever I restart my phone. 
  • The execution of the pending intent will be completely unnoticeable unless you display something. For myself, I decided to show a notification indicating which method was fired, or if there was a failure.
The code is hosted on btibucket.

If you have any questions, please leave a comment and I will definitely respond.

Friday, November 20, 2015

Comskip: For my next trick, I'm going to make these commercials dissapear

For a couple months now I've been recording live tv as in comes in through my Over the air antenna with the help of my TabloTV. Check out this blog post to see how I copy the recorded shows from the tablo to my plex server.

But the idea came to me to take it a step further. Although it's nice that I can record these shows and watch them at my leisure (and even fast forward through any parts i want to) it's still pretty annoying to have to fast forward through the commercials.



As I searched around for a tool to fix this, I found several programs such as Avidemux that allow you to remove commercials (or any part) with relative ease and speed. I'm still not sure if Avidemux is a play on Avid (the sophisticated editing software) or A vide'o mux'er. Either way, there was one problem with these tools -- they were GUI based, which meant that automation would be difficult if not impossible.



Finally I found a tool that could also be used in the terminal -- Comskip. Comskip uses cues in the video such as black frames, audio silence/changes, and more to detect when commercials start. Then it outputs an ".edl" file which stands for edit decision list. The linux version isn't as well distributed as the windows version so I had to compile it from source and download some dependencies.
This page is where i started.

It requires ffmpeg 0.9 or greater and argtable2.

cd /var/tmp
mkdir ComSkipWork
wget http://ffmpeg.org/releases/ffmpeg-2.2.tar.bz2
tar xf ffmpeg-2.2.tar.bz2
cd ffmpeg-2.2
./configure --prefix=/var/tmp/ComSkipWork/install
make -j8
make install
cd ..
wget http://www.xilka.com/xilka/source/comskip-0.93i.tar.xz
tar xf comskip-0.93i.tar.xz
cd comskip-0.93i
PKG_CONFIG_PATH=/var/tmp/ComSkipWork/install/lib/pkgconfig ./configure --prefix=/usr
make -j8
sudo make install


Following the above steps line for line worked perfectly except for one spot



Running make -j8 shows an error about yasm not being installed. Simply run
sudo apt-get install yasm
**If you have other problems see the bottom of this post. I actually didn't start with the above steps and therefore ended up installing a lot things on my own which may or may not be necessary.**

Now that I had comskip installed, it was time to create/setup an .ini file.
A quick google search also revealed that some have customized their .ini files for different regions.

I downloaded USA_General by Wilky13.

For my purposes I just needed to make sure that it would produce an .edl file. After confirming it that, I could now run:
comskip "filename"
and get an .edl file which tells where the commercials are.



So using the .edl file I wrote a python script to scan for any raw videos (which in my case are always .mp4 files) and compress them into .mkv files. From there it analyzes the mkv file and creates multiple temp files of the content between commercial breaks -- or the actual show. The script then concatenates these temp files into a new mkv and moves the compressed version to a backup folder in case there was a mistake. Then it cleans up all the temp files and scans for the next raw mp4. Running from a cron job every night, the OTA shows goes all the way from being recorded to the tablo, copied to my hard drive, compressed and spliced, then loaded into plex automatically.

The code is a little sloppy, but one needs to simply modify the constants including path to a static ffmpeg, and the root directory of where all their tv shows are kept.
For example, on my server



I simply point to /../TabloTv/tv and all folders and sub folders in tv will be scanned.

Here's the python script.

usage:
./removeCommercials.py
(scans all folders and subfolders found in constants.TV_FOLDER)

or

./removeCommercials.py /path/to/filename
(executes script against passed in video only)

removeCommercials.zip


#cordCutter



TROUBLESHOOTING

My first attempt followed these similar steps (i already had a static build of ffmpeg so i thought i wouldn't need it here--wrong!!), but ended with much different results.

cd /var/tmp
mkdir ComSkipWork
wget http://www.xilka.com/xilka/source/comskip-0.93i.tar.xz
tar xf comskip-0.93i.tar.xz
cd comskip-0.93i
./configure --prefix=/usr
make -j8
sudo make install

Right about here
./configure --prefix=/usr
you might hit a error like this:
configure: error: *** A compiler with support for C++11 language features is required.
Ubuntu comes with the g++ compiler, but doesn't seem to cut it. Go ahead and run
sudo apt-get install g++
Now you might hit this issue:
 No package 'libavcodec' found
Another simple fix
 sudo apt-get install libavcodec-dev
And while you're at it
 sudo apt-get install libavformat-dev
 sudo apt-get install libargtable2-dev