Is there a way to transpose data in Hive?

I don't know of a way out of the box in hive to do this, sorry. You get close with explode etc. but I don't think it can get the job done.

Overall, conceptually, I think it's hard to a transpose without knowing what the columns of the destination table are going to be in advance. This is true, in particular for hive, because the metadata related to how many columns, their types, their names, etc. in a database - the metastore. And, it's true in general, because not knowing the columns beforehand, would require some sort of in-memory holding of data (ok, sure with spills) and users may need to be careful about not overflowing the memory and such (just like dynamic partitioning in hive).

In any case, long story short, if you know the columns of the destination table beforehand, life is good. There isn't a set command in hive per se, to the best of my knowledge, but you could use a bunch of if clauses and case statements (ugly I know, but that's how I have done the same in the past) in the select clause to transpose the data. Something along the lines of SQL - How to transpose?

Do let me know how it goes!


As Mark pointed out there's no easy way to do this in Hive since PIVOT doesn't present in Hive and you may also encounter issues when trying to use the case/when 'trick' since you have multiple values (proc1,proc2,proc3).

As for testing purposes, you may try a different approach:

select v, o1, o2, o3 from (
  select k, 
         v,
         LEAD(v,3) OVER() as o1,
         LEAD(v,6) OVER() as o2,
         LEAD(v,9) OVER() as o3
  from (select transform(name,proc1,proc2,proc3) using 'python strm.py' AS (k, v) 
    from input_table) q1
) q2 where k = 'A1';

where strm.py:

import sys

for line in sys.stdin:
  line = line.strip()
  name, proc1, proc2, proc3 = line.split('\t')
  print '%s\t%s' % (name, proc1)
  print '%s\t%s' % (name, proc2)
  print '%s\t%s' % (name, proc3)

The trick here is to use a python script in the map phase which emits each column of a row as distinct rows. Then every third (since we have 3 proc columns) row will form the resulting row which we get by peeking forward (lead).

However, this query does the job, it has the drawback that as the input grows, you need to peek the next 3rd element in the query which may lead to performance hit. Anyway you may evaluate it for testing purposes.