Overcome AD/LDAP limitations

[gwaitsi]

We have about 1500 users in different contained and i have successfully authenticated the users to AD/LDAP and filtered to only the active users showing up.

My problem is, i don't get all the users i.e. only to "P" because of what appears to be the AD MaxPageSize setting of 1000.

my admin guys quoted the following from technet and don't want to increase the setting.
"The LDAP administrative limits balance the Active Directory operational capabilities and its performance. These limits prevent specific operations from adversely affecting the performance of the server, and also make the server resilient to denial of service attacks. Increasing this setting beyond its default value could have an adverse impact on your Active Directory infrastructure"

As we have 4 or 5 users containers, is it possible to build the customer/user list from multiple lower level searches rather than one higher level search?

[gwaitsi]

After further reading, i understand the queries should used results control per below. Is it possible to configure OTRS to use paged results for the customers/users list?


The simple paged results control is a type of control that can be attached to a search operation to indicate that only a subset of the results should be returned. It may be used to iterate through the search results a page at a time. It is similar to the virtual list view control with the exception that it doesn't require the results to be sorted and can only be used to iterate sequentially through the search results.

This control may be useful when the LDAP client has limited resources, on very large resultSetss where the client may not be able to process the entire result set from a given LDAP query, or when the LDAP client is connected over a low-bandwidth connection.

The simple paged results control is defined in RFC 2696. The same control is used in both the search request and search result done messages. It has an OID of 1.2.840.113556.1.4.319, and the value is encoded as follows:

realSearchControlValue ::= SEQUENCE {
size INTEGER (0..maxInt),
-- requested page size from client
-- result set size estimate from server
cookie OCTET STRING
}

Defined In RFC-2696

The Simple Paged Result Control can be used to access all of a large results set when there is a server-side administrative limit to the number of items returned from a query.

For example, Active Directory servers have a default server-side limit of 1000 entries as the maximum number of results that are returned in a single request. If the results of a query exceed this limit, the Paged Results control is used with a page size equal to or less than the server-side limit in order to retrieve all of the results of the query.

[crythias]

LDAP is simply queried for customer information on demand for one/per customer, so the limitation is only because you're looking at the list. Are you encountering information that counters this statement?

[gwaitsi]

If i use "Customers" from the backend, i only receive a partial list of customers/users - we have approx 1500. i.e. we only receive up to the letter "P".
In the log is the error "Size Limit Exceeded". This also occurs if i use an LDAP browser. We have Win2003 and i understand the MaxPageSize is 1000.

[crythias]

There's a difference between the equivalent of

SELECT id, username, firstname, lastname FROM userlist LIMIT 1000 // more or less, what you're seeing
and
SELECT id, username, firstname, lastname FROM userlist WHERE id=1200 //more or less, what OTRS actually does

I know, it's not exactly the same thing, but the point is, the data's still there, I only need to get one of them.

Again, it's a lookup (Query), not a copy or sync of the database.

[gwaitsi]

I changed the below setting to a range of numbers between 1500 and 4000 (current) - doesn't make any difference to where it stops in my list.

CustomerUserSearchListLimit => 250,

[crythias]

You will not be able to overcome this LDAP server-side limitation. Nothing is broken.

[gwaitsi]

no, but from my understanding, the application should build its table using paging per my earlier post, then this wouldn't be an issue.

[crythias]

The application doesn't build a table (in the database). It queries data on demand.

[gwaitsi]

"The simple paged results control is a type of control that can be attached to a search operation to indicate that only a subset of the results should be returned. It may be used to iterate through the search results a page at a time. It is similar to the virtual list view control with the exception that it doesn't require the results to be sorted and can only be used to iterate sequentially through the search results."

page control has nothing to do with building a database. did you not read my second post on this topic. clearly, there should be some page control built into the search as i am sure i would not be the only one effected by directory search limitations on the server side.

[crythias]

If paged results doesn't exist, it'll need to be an enhancement and/or bug request, or request custom development.

I doubt anyone is negatively affected by search limitations because most of the time, you're only addressing one person, and I'm pretty certain if you wanted to, you could search for q* and get all the q users. But since you can't do anything (read: you cannot select, modify, or delete) in bulk on the customer search results screen anyway, and since OTRS doesn't store LDAP data, ... what is the concern? That you can't see it in a big list or that you can't access q-z? If the latter, this is a horrible bug, if the former, what can I say? You're right, and if someone wanted to get to page 17 of 20000 ldap entries, 1000 at a time, your experience would probably apply to that person as well. For access and making changes one user at a time, it won't matter that you couldn't see 20,000 entries.

[gwaitsi]

Well, when you click on "Customers" from the backend menu,
in my case it takes one or two minutes to build the list of people because it has to traverse the whole tree (we have users and different country containers).

From the list that is built, i know there must be 200+ missing (although i admit not having accounted the full 1000 users) - maybe i should.
Every time an agent clicks on customers, it generates a size limit error in the log.

yes, you are right. you can use "r*" for example for a new search and get access to all customers that way. as i see it, there therefore creates a couple choices;

a) enable building the initial customer list from multiple searches i.e. for when more than one user container exists

b) use page filtered searches for building the initial customer list to ensure server limits don't cause a problem

c) use a page limit on the initial customer page to display only an initial number of enteries. thus speeding the building of the initial customer list.

[crythias]

Now that makes sense, or somehow change the initial list to be "25" or so entries just to show some data.

[gwaitsi]

yes, it seems i was fighting myself mixing the two things.

I have set
CustomerUserSearchListLimit => 100,

which gives me a quick response and then the ability to filter for specific names. so functionally it works.

Nevertheless, i still get a size limit exceeded because i have more than 1000 users and the OTRS search is not page search per earlier discussions.
In summary, you are correct. this should be a feature request/bug fix - because the search should be able to deal with server side limitations such as when integrated with AD.

Thanks for challenging the brain. i was trying to use the CustomerUserSearchListLimit to display the full list of customers, which was the wrong approach.