PKCS#11 Test suite (PIV)

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

PKCS#11 Test suite (PIV)

Jakub Jelen
Hello OpenSC devels,

I didn't find any test suite or unit tests for OpenSC project. As I
noticed, there is a lot of hand-testing work on pull requests for
various cards and users. I believe everyone has some use cases to verify
basic functionality of their cards.
I understand that this fields is very divergent, there is a lot of card
variants and it is almost impossible to build automatic test suite that
would run in cloud with every build. But would it make sense to have
something that devels (or users) can simply run and what would verify
basic functionality and possible regressions?

I went to the directory src/tests/ and fixed the tests that are
available now (see pull request [1], broken for 6 years), but they are
far away from complete test suite.

I also started with the idea from PKCS#11 API and put together basic
test suite and inspector for OpenSC, which is currently in my OpenSC
fork [2]. It is by no mean complete test suite of all the use cases, but
I tried to catch most common cases, represent results in understandable
form (currently tested with PIV cards) and add regression test for
recent pull request [3].

And there is the twist. What would you expect from PKCS#11/Smartcard
testsuite? Would it make sense to have something like this upstream?
What use cases would you expect from that to check?


[1] https://github.com/OpenSC/OpenSC/pull/759
[2] https://github.com/Jakuje/OpenSC/tree/jjelen-testsuite/src/tests
[3] https://github.com/OpenSC/OpenSC/pull/743

Regards,

--
Jakub Jelen
Security Technologies
Red Hat


------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Opensc-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opensc-devel
Reply | Threaded
Open this post in threaded view
|

Re: PKCS#11 Test suite (PIV)

Andreas Schwier (ML)
Dear Jakub,

having a central regression test set-up would be great thing.

Internally we use a PKCS#11 test suite similar to [1] that validates the
PKCS#11 interface in combination with our SmartCard-HSM. We also have a
smoke-test that runs a couple of tests on the command line.

Other than that we occasionally do regression testing of the Minidriver.

I guess the biggest issue is, that you need a specific configuration for
each card. If there was a common testing framework, I'd be happy to
provide tests for our card.

Andreas


[1] https://github.com/CardContact/sc-hsm-embedded/tree/master/p11tests

On 05/16/2016 05:04 PM, Jakub Jelen wrote:

> Hello OpenSC devels,
>
> I didn't find any test suite or unit tests for OpenSC project. As I
> noticed, there is a lot of hand-testing work on pull requests for
> various cards and users. I believe everyone has some use cases to verify
> basic functionality of their cards.
> I understand that this fields is very divergent, there is a lot of card
> variants and it is almost impossible to build automatic test suite that
> would run in cloud with every build. But would it make sense to have
> something that devels (or users) can simply run and what would verify
> basic functionality and possible regressions?
>
> I went to the directory src/tests/ and fixed the tests that are
> available now (see pull request [1], broken for 6 years), but they are
> far away from complete test suite.
>
> I also started with the idea from PKCS#11 API and put together basic
> test suite and inspector for OpenSC, which is currently in my OpenSC
> fork [2]. It is by no mean complete test suite of all the use cases, but
> I tried to catch most common cases, represent results in understandable
> form (currently tested with PIV cards) and add regression test for
> recent pull request [3].
>
> And there is the twist. What would you expect from PKCS#11/Smartcard
> testsuite? Would it make sense to have something like this upstream?
> What use cases would you expect from that to check?
>
>
> [1] https://github.com/OpenSC/OpenSC/pull/759
> [2] https://github.com/Jakuje/OpenSC/tree/jjelen-testsuite/src/tests
> [3] https://github.com/OpenSC/OpenSC/pull/743
>
> Regards,
>


--

    ---------    CardContact Systems GmbH
   |.##> <##.|   Schülerweg 38
   |#       #|   D-32429 Minden, Germany
   |#       #|   Phone +49 571 56149
   |'##> <##'|   http://www.cardcontact.de
    ---------    Registergericht Bad Oeynhausen HRB 14880
                 Geschäftsführer Andreas Schwier

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Opensc-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opensc-devel
Reply | Threaded
Open this post in threaded view
|

Re: PKCS#11 Test suite (PIV)

Douglas E Engert
In reply to this post by Jakub Jelen


On 5/16/2016 10:04 AM, Jakub Jelen wrote:

> Hello OpenSC devels,
>
> I didn't find any test suite or unit tests for OpenSC project. As I
> noticed, there is a lot of hand-testing work on pull requests for
> various cards and users. I believe everyone has some use cases to verify
> basic functionality of their cards.
> I understand that this fields is very divergent, there is a lot of card
> variants and it is almost impossible to build automatic test suite that
> would run in cloud with every build. But would it make sense to have
> something that devels (or users) can simply run and what would verify
> basic functionality and possible regressions?

In-cloud testing would require the cloud test machine to have physical cards.
(Unless you are suggesting some RDC access to cards.)

I would say the closest tool we have is: pkcs11-tool -t -l
It does some basic tests, but as you may have noted if you try and run it
with a PIV card, it has some problems, especially with the decryption, as it
says the user is not logged in when trying to use the Sign key. The key usage
says it should not be used for decryption. With other cards it may have different problems.

>
> I went to the directory src/tests/ and fixed the tests that are
> available now (see pull request [1], broken for 6 years), but they are
> far away from complete test suite.
>
> I also started with the idea from PKCS#11 API and put together basic
> test suite and inspector for OpenSC, which is currently in my OpenSC
> fork [2]. It is by no mean complete test suite of all the use cases, but
> I tried to catch most common cases, represent results in understandable
> form (currently tested with PIV cards) and add regression test for
> recent pull request [3].

Good choice of card :-)

Are you using the the NIST set of 16 demo/test cards?

>
> And there is the twist. What would you expect from PKCS#11/Smartcard
> testsuite? Would it make sense to have something like this upstream?
> What use cases would you expect from that to check?

Enhancing the functionality of the pkcs11-tool -t would be a good start.


>
>
> [1] https://github.com/OpenSC/OpenSC/pull/759
> [2] https://github.com/Jakuje/OpenSC/tree/jjelen-testsuite/src/tests
> [3] https://github.com/OpenSC/OpenSC/pull/743
>
> Regards,
>

--

  Douglas E. Engert  <[hidden email]>


------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Opensc-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opensc-devel
Reply | Threaded
Open this post in threaded view
|

Re: PKCS#11 Test suite (PIV)

Nikos Mavrogiannopoulos-2
On Mon, May 16, 2016 at 7:20 PM, Douglas E Engert <[hidden email]> wrote:

>> I didn't find any test suite or unit tests for OpenSC project. As I
>> noticed, there is a lot of hand-testing work on pull requests for
>> various cards and users. I believe everyone has some use cases to verify
>> basic functionality of their cards.
>> I understand that this fields is very divergent, there is a lot of card
>> variants and it is almost impossible to build automatic test suite that
>> would run in cloud with every build. But would it make sense to have
>> something that devels (or users) can simply run and what would verify
>> basic functionality and possible regressions?
> In-cloud testing would require the cloud test machine to have physical cards.
> (Unless you are suggesting some RDC access to cards.)

Such a test suite has two goals. (1) for the card makers to verify
that their cards run with opensc, and (2) for the opensc developers to
verify that their new code does not regress. The cloud testing is
mainly applicable for (2), and that may be addressed with a card
simulator.

>> And there is the twist. What would you expect from PKCS#11/Smartcard
>> testsuite? Would it make sense to have something like this upstream?
>> What use cases would you expect from that to check?
> Enhancing the functionality of the pkcs11-tool -t would be a good start.

What would be the advantage of having as part of that tool?

regards,
Nikos

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Opensc-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opensc-devel
Reply | Threaded
Open this post in threaded view
|

Re: PKCS#11 Test suite (PIV)

Jakub Jelen
In reply to this post by Andreas Schwier (ML)
Hello Andreas,

On 05/16/2016 05:40 PM, Andreas Schwier wrote:
> Dear Jakub,
>
> having a central regression test set-up would be great thing.
> Internally we use a PKCS#11 test suite similar to [1] that validates the
> PKCS#11 interface in combination with our SmartCard-HSM. We also have a
> smoke-test that runs a couple of tests on the command line.
Wow, PKCS#11 test suite in Javascript. I did not see everything yet. But
the concept looks similar to the think that we put together. Having
something similar in OpenSC that would be possible to run by as many
users with as many cards as possible would be ultimate goal.

Basically what I see in the linked testsuite is:
  * initialization, key and certificate generation
    * (not applicable for test PIV cards I have)
    * card dependent
  * enumerate
  * encrypt
  * decrypt
  * signature (with and without digest)
    * should be card independent, isn't it?
> Other than that we occasionally do regression testing of the Minidriver.
>
> I guess the biggest issue is, that you need a specific configuration for
> each card. If there was a common testing framework, I'd be happy to
> provide tests for our card.
Great to hear it.
> Andreas
>
>
> [1] https://github.com/CardContact/sc-hsm-embedded/tree/master/p11tests
Thanks for reference and ideas.
Regards,

--
Jakub Jelen
Security Technologies
Red Hat


------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Opensc-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opensc-devel
Reply | Threaded
Open this post in threaded view
|

Re: PKCS#11 Test suite (PIV)

Jakub Jelen
In reply to this post by Douglas E Engert
On 05/16/2016 07:20 PM, Douglas E Engert wrote:

> On 5/16/2016 10:04 AM, Jakub Jelen wrote:
>> Hello OpenSC devels,
>>
>> I didn't find any test suite or unit tests for OpenSC project. As I
>> noticed, there is a lot of hand-testing work on pull requests for
>> various cards and users. I believe everyone has some use cases to verify
>> basic functionality of their cards.
>> I understand that this fields is very divergent, there is a lot of card
>> variants and it is almost impossible to build automatic test suite that
>> would run in cloud with every build. But would it make sense to have
>> something that devels (or users) can simply run and what would verify
>> basic functionality and possible regressions?
> In-cloud testing would require the cloud test machine to have physical cards.
> (Unless you are suggesting some RDC access to cards.)
Yes, this would be awesome, but probably impossible to achieve. Rather
having
something that can be simply run by many users with many different cards and
report general success or failure would be achievable.

Theoretically generating logs, collecting them from different cards and
users and
representing them in readable form could be too useful for future
codebase stability
among releases. But it is a bit over the initial idea.
> I would say the closest tool we have is: pkcs11-tool -t -l
> It does some basic tests, but as you may have noted if you try and run it
> with a PIV card, it has some problems, especially with the decryption, as it
> says the user is not logged in when trying to use the Sign key. The key usage
> says it should not be used for decryption. With other cards it may have different problems.
Thanks for mentioning pkcs11-tool test mode. I struggled upon it, but
there were
some problems that prevented me to work with that. I will check what can
be done there.

>> I went to the directory src/tests/ and fixed the tests that are
>> available now (see pull request [1], broken for 6 years), but they are
>> far away from complete test suite.
>>
>> I also started with the idea from PKCS#11 API and put together basic
>> test suite and inspector for OpenSC, which is currently in my OpenSC
>> fork [2]. It is by no mean complete test suite of all the use cases, but
>> I tried to catch most common cases, represent results in understandable
>> form (currently tested with PIV cards) and add regression test for
>> recent pull request [3].
> Good choice of card :-)
>
> Are you using the the NIST set of 16 demo/test cards?
Yes.

Regards,

--
Jakub Jelen
Security Technologies
Red Hat


------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Opensc-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opensc-devel
Reply | Threaded
Open this post in threaded view
|

Re: PKCS#11 Test suite (PIV)

Thomas Calderon
Hi everybody,

In addition to pkcs11-tool it might be worth mentioning opkcs11-tool [1], a tool that I co-develop.

It mimics the CLI interface of pkcs11-tool but allows for advanced PKCS#11 use cases:
  - template based operations (management and crypto)
  - PSS signature
  - Wrap/Unwrap
  - and much more as it is easy to extend

In a sense, the tool could be used to perform some regression testing from the CLI.


Cheers,

Thomas

On Tue, May 17, 2016 at 1:39 PM, Jakub Jelen <[hidden email]> wrote:
On 05/16/2016 07:20 PM, Douglas E Engert wrote:
> On 5/16/2016 10:04 AM, Jakub Jelen wrote:
>> Hello OpenSC devels,
>>
>> I didn't find any test suite or unit tests for OpenSC project. As I
>> noticed, there is a lot of hand-testing work on pull requests for
>> various cards and users. I believe everyone has some use cases to verify
>> basic functionality of their cards.
>> I understand that this fields is very divergent, there is a lot of card
>> variants and it is almost impossible to build automatic test suite that
>> would run in cloud with every build. But would it make sense to have
>> something that devels (or users) can simply run and what would verify
>> basic functionality and possible regressions?
> In-cloud testing would require the cloud test machine to have physical cards.
> (Unless you are suggesting some RDC access to cards.)
Yes, this would be awesome, but probably impossible to achieve. Rather
having
something that can be simply run by many users with many different cards and
report general success or failure would be achievable.

Theoretically generating logs, collecting them from different cards and
users and
representing them in readable form could be too useful for future
codebase stability
among releases. But it is a bit over the initial idea.
> I would say the closest tool we have is: pkcs11-tool -t -l
> It does some basic tests, but as you may have noted if you try and run it
> with a PIV card, it has some problems, especially with the decryption, as it
> says the user is not logged in when trying to use the Sign key. The key usage
> says it should not be used for decryption. With other cards it may have different problems.
Thanks for mentioning pkcs11-tool test mode. I struggled upon it, but
there were
some problems that prevented me to work with that. I will check what can
be done there.
>> I went to the directory src/tests/ and fixed the tests that are
>> available now (see pull request [1], broken for 6 years), but they are
>> far away from complete test suite.
>>
>> I also started with the idea from PKCS#11 API and put together basic
>> test suite and inspector for OpenSC, which is currently in my OpenSC
>> fork [2]. It is by no mean complete test suite of all the use cases, but
>> I tried to catch most common cases, represent results in understandable
>> form (currently tested with PIV cards) and add regression test for
>> recent pull request [3].
> Good choice of card :-)
>
> Are you using the the NIST set of 16 demo/test cards?
Yes.

Regards,

--
Jakub Jelen
Security Technologies
Red Hat


------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Opensc-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opensc-devel


------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Opensc-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opensc-devel
Reply | Threaded
Open this post in threaded view
|

Re: PKCS#11 Test suite (PIV)

Jakub Jelen
In reply to this post by Jakub Jelen
Hi,
few more notes.

On 05/17/2016 02:39 PM, Jakub Jelen wrote:

> On 05/16/2016 07:20 PM, Douglas E Engert wrote:
>> I would say the closest tool we have is: pkcs11-tool -t -l
>> It does some basic tests, but as you may have noted if you try and run it
>> with a PIV card, it has some problems, especially with the decryption, as it
>> says the user is not logged in when trying to use the Sign key. The key usage
>> says it should not be used for decryption. With other cards it may have different problems.
> Thanks for mentioning pkcs11-tool test mode. I struggled upon it, but
> there were
> some problems that prevented me to work with that. I will check what can
> be done there.
The  pkcs11-tool -t  has most of the problems with PIV cards I was
solving during last weeks:
  * missing ECDSA mechanisms support (more work)
  * C_Verify is not implemented in these cards
  * CKA_ALWAYS_AUTHENTICATE is not respected for decryption (well
implemented for signing)

The first is expected, the second is soft fail and the last one is hard
fail letting down whole test suite
(but should be quite easy fix). Otherwise it supports most of the
general tests, but bundling inside
card-specific initializations or bunches of regression tests
(theoretically) does not seem like a way
to go.

Kind regards,

--
Jakub Jelen
Security Technologies
Red Hat


------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Opensc-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opensc-devel
Reply | Threaded
Open this post in threaded view
|

Re: PKCS#11 Test suite (PIV)

David Woodhouse
In reply to this post by Thomas Calderon
On Tue, 2016-05-17 at 14:51 +0100, Thomas Calderon wrote:
>
> In addition to pkcs11-tool it might be worth mentioning opkcs11-tool
> [1], a tool that I co-develop.

Hm, I note the README on Github doesn't seem to indicate that it
supports RFC7512 PKCS#11 URIs to specify objects. Does it?

The examples also explicitly specify a module, rather than using the
system (typically p11-kit) configuration and just doing the right
thing...

--
dwmw2



------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Opensc-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opensc-devel

smime.p7s (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: PKCS#11 Test suite (PIV)

Thomas Calderon
Hi David,

You are correct opkcs11-tool does not currently support RFC7512, I'll give it a look.
Should be easy enough to add.

That being said, opkcs11-tool was developed to allow fine-grained token management and crypto operations using the CLI, not primarily to leverage system-wide configuration using p11-kit.
Of course, being an open source project, contributions that can speed up this feature are welcome :-)

Cheers,

Thomas


On Tue, May 17, 2016 at 3:53 PM, David Woodhouse <[hidden email]> wrote:
On Tue, 2016-05-17 at 14:51 +0100, Thomas Calderon wrote:
>
> In addition to pkcs11-tool it might be worth mentioning opkcs11-tool
> [1], a tool that I co-develop.

Hm, I note the README on Github doesn't seem to indicate that it
supports RFC7512 PKCS#11 URIs to specify objects. Does it?

The examples also explicitly specify a module, rather than using the
system (typically p11-kit) configuration and just doing the right
thing...

--
dwmw2




------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Opensc-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opensc-devel
Reply | Threaded
Open this post in threaded view
|

Re: PKCS#11 Test suite (PIV)

Nikos Mavrogiannopoulos-2
In reply to this post by Andreas Schwier (ML)
On Mon, May 16, 2016 at 5:40 PM, Andreas Schwier
<[hidden email]> wrote:
> Dear Jakub,
> having a central regression test set-up would be great thing.
> Internally we use a PKCS#11 test suite similar to [1] that validates the
> PKCS#11 interface in combination with our SmartCard-HSM. We also have a
> smoke-test that runs a couple of tests on the command line.
[...]
> I guess the biggest issue is, that you need a specific configuration for
> each card. If there was a common testing framework, I'd be happy to
> provide tests for our card.

The test suite Jakub provided is read-only in nature and can be shared
by several cards.
Would it make sense to split the testing to:
(initialization part) - (write params part) - (read part)

Where the write/initialization part can be unique per card and can
simply be a script calling other tools. The write part generates some
test parameters (certificates and keys) under some criteria we agree
on and is shared between families of cards (ECDSA/RSA). Finally, the
read only part verifies the correct operation with the parameters
written.

Each part can be a different tool, and can even be a pkcs11-tool
option, and can be combined under a per-card specific script (the
initialization). For example:
mycard.sh:
  initialize_that_card_with_pin 1234
  run_write_params_test_with_pin 1234
  run_read_params_test_with_pin 1234
  clear_card

For the PIV cards the test suite is written on it would be simply a
run_read_params_test_with_pin call.

Would that be some framework you'd like to provide tests under? For
your cards for example, the initialization part needs to be written,
and followed up by the read-only check.

regards,
Nikos

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Opensc-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opensc-devel
Reply | Threaded
Open this post in threaded view
|

Re: PKCS#11 Test suite (PIV)

Douglas E Engert


On 5/19/2016 2:07 AM, Nikos Mavrogiannopoulos wrote:

> On Mon, May 16, 2016 at 5:40 PM, Andreas Schwier
> <[hidden email]> wrote:
>> Dear Jakub,
>> having a central regression test set-up would be great thing.
>> Internally we use a PKCS#11 test suite similar to [1] that validates the
>> PKCS#11 interface in combination with our SmartCard-HSM. We also have a
>> smoke-test that runs a couple of tests on the command line.
> [...]
>> I guess the biggest issue is, that you need a specific configuration for
>> each card. If there was a common testing framework, I'd be happy to
>> provide tests for our card.
>
> The test suite Jakub provided is read-only in nature and can be shared
> by several cards.
> Would it make sense to split the testing to:
> (initialization part) - (write params part) - (read part)


Yes. Many people using this test suite may be using their national ID or employer
issued smart card. They do not know the PUK, or other keys needed to write to the card.
Only the read part would be run. Also keep in mind that if any test fails with a bad pin,
no other tests should try the pin again to avoid locking the card.

For ECDH derive, 2 cards are needed. Each card needs as input the public
key of the other. A software ephemeral key could be used in place of the second card.


>
> Where the write/initialization part can be unique per card and can
> simply be a script calling other tools. The write part generates some
> test parameters (certificates and keys) under some criteria we agree
> on and is shared between families of cards (ECDSA/RSA). Finally, the
> read only part verifies the correct operation with the parameters
> written.
>
> Each part can be a different tool, and can even be a pkcs11-tool
> option, and can be combined under a per-card specific script (the
> initialization). For example:
> mycard.sh:
>   initialize_that_card_with_pin 1234
>   run_write_params_test_with_pin 1234
>   run_read_params_test_with_pin 1234
>   clear_card
>
> For the PIV cards the test suite is written on it would be simply a
> run_read_params_test_with_pin call.

The NEO piv tool and the OpenSC piv-tool can do some initialization and write operations.
But each card vendor has different options, and each batch of cards is supplied
with a 3DES or AES key needed to allow writing to the card.
Other cards also have <card>-tool programs.

>
> Would that be some framework you'd like to provide tests under? For
> your cards for example, the initialization part needs to be written,
> and followed up by the read-only check.
>
> regards,
> Nikos
>
> ------------------------------------------------------------------------------
> Mobile security can be enabling, not merely restricting. Employees who
> bring their own devices (BYOD) to work are irked by the imposition of MDM
> restrictions. Mobile Device Manager Plus allows you to control only the
> apps on BYO-devices by containerizing them, leaving personal data untouched!
> https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
> _______________________________________________
> Opensc-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/opensc-devel
>

--

  Douglas E. Engert  <[hidden email]>


------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Opensc-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/opensc-devel