[[_custom_importer]]
==== A Custom Importer

(((git commands, fast-import)))
(((Importing, from others)))
If your system isn't one of the above, you should look for an importer online – quality importers are available for many other systems, including CVS, Clear Case, Visual Source Safe, even a directory of archives.
If none of these tools works for you, you have a more obscure tool, or you otherwise need a more custom importing process, you should use `git fast-import`.
This command reads simple instructions from stdin to write specific Git data.
It's much easier to create Git objects this way than to run the raw Git commands or try to write the raw objects (see <<ch10-git-internals#ch10-git-internals>> for more information).
This way, you can write an import script that reads the necessary information out of the system you're importing from and prints straightforward instructions to stdout.
You can then run this program and pipe its output through `git fast-import`.

To quickly demonstrate, you'll write a simple importer.
Suppose you work in `current`, you back up your project by occasionally copying the directory into a time-stamped `back_YYYY_MM_DD` backup directory, and you want to import this into Git.
Your directory structure looks like this:

[source,console]
----
$ ls /opt/import_from
back_2014_01_02
back_2014_01_04
back_2014_01_14
back_2014_02_03
current
----

In order to import a Git directory, you need to review how Git stores its data.
As you may remember, Git is fundamentally a linked list of commit objects that point to a snapshot of content.
All you have to do is tell `fast-import` what the content snapshots are, what commit data points to them, and the order they go in.
Your strategy will be to go through the snapshots one at a time and create commits with the contents of each directory, linking each commit back to the previous one.

As we did in <<ch08-customizing-git#_an_example_git_enforced_policy>>, we'll write this in Ruby, because it's what we generally work with and it tends to be easy to read.
You can write this example pretty easily in anything you're familiar with – it just needs to print the appropriate information to `stdout`.
And, if you are running on Windows, this means you'll need to take special care to not introduce carriage returns at the end your lines – `git fast-import` is very particular about just wanting line feeds (LF) not the carriage return line feeds (CRLF) that Windows uses.

To begin, you'll change into the target directory and identify every subdirectory, each of which is a snapshot that you want to import as a commit.
You'll change into each subdirectory and print the commands necessary to export it.
Your basic main loop looks like this:

[source,ruby]
----
last_mark = nil

# loop through the directories
Dir.chdir(ARGV[0]) do
  Dir.glob("*").each do |dir|
    next if File.file?(dir)

    # move into the target directory
    Dir.chdir(dir) do
      last_mark = print_export(dir, last_mark)
    end
  end
end
----

You run `print_export` inside each directory, which takes the manifest and mark of the previous snapshot and returns the manifest and mark of this one; that way, you can link them properly.
"`Mark`" is the `fast-import` term for an identifier you give to a commit; as you create commits, you give each one a mark that you can use to link to it from other commits.
So, the first thing to do in your `print_export` method is generate a mark from the directory name:

[source,ruby]
----
mark = convert_dir_to_mark(dir)
----

You'll do this by creating an array of directories and using the index value as the mark, because a mark must be an integer.
Your method looks like this:

[source,ruby]
----
$marks = []
def convert_dir_to_mark(dir)
  if !$marks.include?(dir)
    $marks << dir
  end
  ($marks.index(dir) + 1).to_s
end
----

Now that you have an integer representation of your commit, you need a date for the commit metadata.
Because the date is expressed in the name of the directory, you'll parse it out.
The next line in your `print_export` file is:

[source,ruby]
----
date = convert_dir_to_date(dir)
----

where `convert_dir_to_date` is defined as:

[source,ruby]
----
def convert_dir_to_date(dir)
  if dir == 'current'
    return Time.now().to_i
  else
    dir = dir.gsub('back_', '')
    (year, month, day) = dir.split('_')
    return Time.local(year, month, day).to_i
  end
end
----

That returns an integer value for the date of each directory.
The last piece of meta-information you need for each commit is the committer data, which you hardcode in a global variable:

[source,ruby]
----
$author = 'John Doe <john@example.com>'
----

Now you're ready to begin printing out the commit data for your importer.
The initial information states that you're defining a commit object and what branch it's on, followed by the mark you've generated, the committer information and commit message, and then the previous commit, if any.
The code looks like this:

[source,ruby]
----
# print the import information
puts 'commit refs/heads/master'
puts 'mark :' + mark
puts "committer #{$author} #{date} -0700"
export_data('imported from ' + dir)
puts 'from :' + last_mark if last_mark
----

You hardcode the time zone (-0700) because doing so is easy.
If you're importing from another system, you must specify the time zone as an offset.
The commit message must be expressed in a special format:

[source]
----
data (size)\n(contents)
----

The format consists of the word data, the size of the data to be read, a newline, and finally the data.
Because you need to use the same format to specify the file contents later, you create a helper method, `export_data`:

[source,ruby]
----
def export_data(string)
  print "data #{string.size}\n#{string}"
end
----

All that's left is to specify the file contents for each snapshot.
This is easy, because you have each one in a directory – you can print out the `deleteall` command followed by the contents of each file in the directory.
Git will then record each snapshot appropriately:

[source,ruby]
----
puts 'deleteall'
Dir.glob("**/*").each do |file|
  next if !File.file?(file)
  inline_data(file)
end
----

Note: Because many systems think of their revisions as changes from one commit to another, fast-import can also take commands with each commit to specify which files have been added, removed, or modified and what the new contents are.
You could calculate the differences between snapshots and provide only this data, but doing so is more complex – you may as well give Git all the data and let it figure it out.
If this is better suited to your data, check the `fast-import` man page for details about how to provide your data in this manner.

The format for listing the new file contents or specifying a modified file with the new contents is as follows:

[source]
----
M 644 inline path/to/file
data (size)
(file contents)
----

Here, 644 is the mode (if you have executable files, you need to detect and specify 755 instead), and inline says you'll list the contents immediately after this line.
Your `inline_data` method looks like this:

[source,ruby]
----
def inline_data(file, code = 'M', mode = '644')
  content = File.read(file)
  puts "#{code} #{mode} inline #{file}"
  export_data(content)
end
----

You reuse the `export_data` method you defined earlier, because it's the same as the way you specified your commit message data.

The last thing you need to do is to return the current mark so it can be passed to the next iteration:

[source,ruby]
----
return mark
----

[NOTE]
====
If you are running on Windows you'll need to make sure that you add one extra step.
As mentioned before, Windows uses CRLF for new line characters while `git fast-import` expects only LF.
To get around this problem and make `git fast-import` happy, you need to tell ruby to use LF instead of CRLF:

[source,ruby]
----
$stdout.binmode
----
====

That's it.
Here's the script in its entirety:

[source,ruby]
----
#!/usr/bin/env ruby

$stdout.binmode
$author = "John Doe <john@example.com>"

$marks = []
def convert_dir_to_mark(dir)
    if !$marks.include?(dir)
        $marks << dir
    end
    ($marks.index(dir)+1).to_s
end

def convert_dir_to_date(dir)
    if dir == 'current'
        return Time.now().to_i
    else
        dir = dir.gsub('back_', '')
        (year, month, day) = dir.split('_')
        return Time.local(year, month, day).to_i
    end
end

def export_data(string)
    print "data #{string.size}\n#{string}"
end

def inline_data(file, code='M', mode='644')
    content = File.read(file)
    puts "#{code} #{mode} inline #{file}"
    export_data(content)
end

def print_export(dir, last_mark)
    date = convert_dir_to_date(dir)
    mark = convert_dir_to_mark(dir)

    puts 'commit refs/heads/master'
    puts "mark :#{mark}"
    puts "committer #{$author} #{date} -0700"
    export_data("imported from #{dir}")
    puts "from :#{last_mark}" if last_mark

    puts 'deleteall'
    Dir.glob("**/*").each do |file|
        next if !File.file?(file)
        inline_data(file)
    end
    mark
end

# Loop through the directories
last_mark = nil
Dir.chdir(ARGV[0]) do
    Dir.glob("*").each do |dir|
        next if File.file?(dir)

        # move into the target directory
        Dir.chdir(dir) do
            last_mark = print_export(dir, last_mark)
        end
    end
end
----

If you run this script, you'll get content that looks something like this:

[source,console]
----
$ ruby import.rb /opt/import_from
commit refs/heads/master
mark :1
committer John Doe <john@example.com> 1388649600 -0700
data 29
imported from back_2014_01_02deleteall
M 644 inline README.md
data 28
# Hello

This is my readme.
commit refs/heads/master
mark :2
committer John Doe <john@example.com> 1388822400 -0700
data 29
imported from back_2014_01_04from :1
deleteall
M 644 inline main.rb
data 34
#!/bin/env ruby

puts "Hey there"
M 644 inline README.md
(...)
----

To run the importer, pipe this output through `git fast-import` while in the Git directory you want to import into.
You can create a new directory and then run `git init` in it for a starting point, and then run your script:

[source,console]
----
$ git init
Initialized empty Git repository in /opt/import_to/.git/
$ ruby import.rb /opt/import_from | git fast-import
git-fast-import statistics:
---------------------------------------------------------------------
Alloc'd objects:       5000
Total objects:           13 (         6 duplicates                  )
      blobs  :            5 (         4 duplicates          3 deltas of          5 attempts)
      trees  :            4 (         1 duplicates          0 deltas of          4 attempts)
      commits:            4 (         1 duplicates          0 deltas of          0 attempts)
      tags   :            0 (         0 duplicates          0 deltas of          0 attempts)
Total branches:           1 (         1 loads     )
      marks:           1024 (         5 unique    )
      atoms:              2
Memory total:          2344 KiB
       pools:          2110 KiB
     objects:           234 KiB
---------------------------------------------------------------------
pack_report: getpagesize()            =       4096
pack_report: core.packedGitWindowSize = 1073741824
pack_report: core.packedGitLimit      = 8589934592
pack_report: pack_used_ctr            =         10
pack_report: pack_mmap_calls          =          5
pack_report: pack_open_windows        =          2 /          2
pack_report: pack_mapped              =       1457 /       1457
---------------------------------------------------------------------
----

As you can see, when it completes successfully, it gives you a bunch of statistics about what it accomplished.
In this case, you imported 13 objects total for 4 commits into 1 branch.
Now, you can run `git log` to see your new history:

[source,console]
----
$ git log -2
commit 3caa046d4aac682a55867132ccdfbe0d3fdee498
Author: John Doe <john@example.com>
Date:   Tue Jul 29 19:39:04 2014 -0700

    imported from current

commit 4afc2b945d0d3c8cd00556fbe2e8224569dc9def
Author: John Doe <john@example.com>
Date:   Mon Feb 3 01:00:00 2014 -0700

    imported from back_2014_02_03
----

There you go – a nice, clean Git repository.
It's important to note that nothing is checked out – you don't have any files in your working directory at first.
To get them, you must reset your branch to where `master` is now:

[source,console]
----
$ ls
$ git reset --hard master
HEAD is now at 3caa046 imported from current
$ ls
README.md main.rb
----

You can do a lot more with the `fast-import` tool – handle different modes, binary data, multiple branches and merging, tags, progress indicators, and more.
A number of examples of more complex scenarios are available in the `contrib/fast-import` directory of the Git source code.
