thanks for the information though.
Kirtan

The techniques I teach can take mappings from 24 hours down to 4 hours, and from 48 hours down to 8 hours, and from 2 hours down to 35 to 45 minutes.

I’ve added v8 content, but all of the content applies backwards to v6, v7 as well.

If you can get 10 people into a class, we can bring the class to you (within the USA). If you can get 20 people in to the class, we can bring the class to you outside the USA.

If you are interested in holding such a class, please let me know. I’ve been teaching performance and tuning (systems-wide, including Informatica) for 15 years.

The class itinerary is available at: http://www.GeneseeAcademy.com

Thanks,
Dan Linstedt
DanL@RapidACE.com
http://www.RapidACE.com

Overall I do agree with your comment though. If an outer join is possible to perform while keeping the cost low, there is nothing better than that.

Kirtan

We have very similar issue as you have, here is what we plan to do:

If you can remove lookup in your mapping and including this in your source qualify sql override,
in our case, (use lookup table outjoin with souce table). our mapping performance increase from 15 mins down to 35 secs, if you added have properly indexes.

Hope this will be help you as well.

What do you expect from this query? Do you have a time limit that you want it to finish by? Or you just think that since it’s taking 35 minutes, it’s slow.

***********Anyways 

Most of the time is taken by the sorting. The problem here is that you are doing a look up on a big table. Doing a lookup against such table would be a big No-No in my opinion. Is TABLE_DETAIL in a different database than the source tables? I have a suggestion for you. See if it works for you.

Let’s say you have a source table called SRC_DATA fields like

A
B
C
DETAIL_ID
NUMBER
YEAR

And than there is your lookup table with fields
DETAIL_ID
NUMBER
YEAR

NOW before running the main process,
GET ALL THE DATA from the lookup table where combination of DETAIL_ID, NUMBER is in
(
get a distinct set of DETAIL_ID, NUMBER
)
And build a view with the results.

And in your main process do a look up on the view and not on the big table.

#####
HOWEVER-
If you HAVE TO do the look up than I would suggest creating an index on all the columns that are in the ORDER BY list. Since you do not have anything in the where clause this is a straight forward. You idea of either creating a better index or altering the ORDER BY clause would work just fine.

ALSO,
Look into tuning look up cache (data and index). That will help you a LOT with your performance issue.
You should be able to find it in the transformation guide.

These are a few thoughts that I gathered while reading your email. If you need a detailed explanation and/or suggestion, please give a test case (create table scripts with 100 rows and a good description of your process (make it technical and not so functional) .

Hope this helps.

-Kirtan

One of our informatica batch is taking a long time(35 min) to execute a sorting query during Look up in a table. The table contains arround 47662364 records. And it takes 19-20 min to execute the below query and thats why it is creating problem in performance. Can we tune this query/ Table so that it takes less time ? There are other similar cases too where the query takes a long time to execute. The query :-
SELECT DETAIL_ID,NUMBER,YEAR FROM TABLE_DETAIL ORDER BY NUMBER,YEAR,DETAIL_ID
Is there any way to tune the table/query. I mean as far as my knowledge goes we can tune this in 2 ways
1. Creating and Using an unique Index on these 3 fields (DETAIL_ID,NUMBER,MVYEAR) on the table.
there is already an index on DETAIL_ID,NUMBER.
2. Creating some tuning method provided by Informatica :-
a. change the default query like
SELECT DETAIL_ID,NUMBER,YEAR FROM TABLE_DETAIL ORDER BY NUMBER,YEAR
Here we can see that we have 2 fields instead of 3( in default query) to sort.
Shall this improve performance ? I am not sure that any of above two can improve performance, because the table is too large and we are sorting all records.

What do you think will be effective ? Database Tuning or Session tuning ?

You can have as many sources as you want. If there are different data sources, I would make sure of the following
[
Before I start with that, let me say this.
Pulling data from multiple sources always rings a bell in my head.
Make sure that you pull tables from multiple data sources only if you want join them. It's ok to have multiple pipelines. Make sure you look at all the requirements and performance impacts before going forward with your approach.
]

1. All the data sources that you are pulling data from are related, that is, they have a logical functional(business) relationship.

2. There are keys on tables (which are from different data sources) that can be used to join these tables.

3. To assign connection values in session properties, open the session -> goto properties -> on the left hand side, select the source/target you want to define connection for -> on the right hand side assign connection value.

I am going to post a tutorial on Informatica Basics soon. Keep checking the web site for that.

This is a good basic start for P&T – interesting statements, there are a few hints I’ll add for you. I’m currently writing a new P&T document, very extensive, that will be available (hopefully Q2 next year) – for sale, quite possible it may turn into a small hand-book.

1. If you use a joiner, always use a sorted joiner – the cost of sorting data coming from the RDBMS staging area is frequently less than the cost of building the caches in place. Furthermore, if you are NOT on 64 bit Informatica, or you don’t have unlimited (seemingly unlimited) RAM, you have an upper limit to the caching mechanisms in all the cached objects – including Joiner.
2. If the RDBMS is NOT tuned properly, putting more work into the RDBMS will actually slow things down.
3. Too many Instances of an RDBMS on the SAME MACHINE will actually kill performance rather than help it, and it really is not necessary in order to handle fail-over as many set it up to do.
4. Replacing a Lookup with a SORTED JOINER can improve performance dramatically.
5. The manuals have the formula wrong for the Data / Index Cache settings, even though it’s counter-intuitive, you want 100% of the INDEX cached if you can get it, giving up Data Cache for disk. Why? Because if you can’t access RAM to check for the existence of data you are actually increasing I/O – when you increase I/O you slow performance dramatically. Set your Index caches to twice the size of your data caches to be safe.

Hope these tips help, Watch my site next year for new announcements on performance and tuning.
Dan L