[erlang-questions] Selecting records from Mnesia using partial key

[Frans Schneider]

Yes, I know I had to be more specific, but this was one of those Friday 
afternoon posts looking for an easy answer.

The thing is, looking for applicable policies all combinations of 
attributes from the request have to be tried: i.e. the number of 
combinations becomes an issue. Fetching policies from Mnesia based on 
their attributes is trivial using hashing.

The number of policies in the Mnesia table may be up to 10K, maybe 100K. 
The number of attributes per request, as far as I have seen, varies 
mostly between 4 and 8. The repository has a primary key for the policy 
id and a index key which is the hash of the attribute ids used by that 
policy. The number of requests is high, as high as possible, and come 
from applications making authorization calls. For horizontal scaling, 
the repository should be distributed.

Looking up policies probably should be implemented using some digraph 
type of structure on top of some storage back-end, maybe a graph 
database. I have seen some implementations of that but all lack the 
possibility to remove or make changes to policies after the graph has 
been created without rebuilding the whole graph. With the regular 
changes made to authorizations, making changes to the graph is mandatory.

Maybe somebody did something before like this and I would like some input.

Frans

[1]

On 07/08/2018 04:45 PM, Mikael Pettersson wrote:
> On Fri, Jul 6, 2018 at 1:34 PM, Frans Schneider <fchschneider@REDACTED> wrote:
>> Dear list,
>>
>> Before trying to reinvent the wheel, maybe somebody can point me in the
>> right direction.
>>
>> I have a request with a variable set of attributes [A, B, C...] and a Mnesia
>> table with policies, each applicable for a also variable set of attributes
>> [X, Y, Z...]. Policies may be applicable for the same set of attributes, in
>> which case all these policies have to be evaluated.
>>
>> A policy should be selected if its attributes is the set _or a subset_ of
>> attributes from the request: i.e. if the request has the set of attributes
>> [A, B] than all policies with attributes [A, B], [A] and [B] should be
>> selected.
>>
>> Attributes currently are records holding two strings but could be turned
>> into atoms.
>>
>> Does anybody know a nice way to implement this in Mnesia?
> 
> Without further information about or restrictions to your problem, I guess the
> simplest solution is to use a qlc to query the policies table, and in the qlc
> filter by available attributes.
> 
> This is effectively going to traverse the entire table, so it won't be
> very efficient,
> but it may be enough for your case (we don't know your table size, number of
> attributes, etc).
> 
> If you want something more efficient, you'll need one or more indices, i.e.
> mappings from attributes (or sets of attributes) to applicable policies.  Since
> your input may contain a superset of a policy's attributes, you'll have to
> create it/them yourself rather than relying on mnesia's built-in
> indexing support.
>